DEFENSE 


JOURNAL 


A  Publication  of  the  Defense  Acquisition  University 


Til IV  UNITED  STATES  Ol' AM K ItM'A 

~ JJR’-SSf-KSU  K  11153712  G 

1TON,  IJ.C5. 


aking  Every  Dollar  Count 


-  Improving  Acquisition  Outcomes  - 


2011  Annual 


/  Research  Paper  Competition 


Calculating  Return  on  Investment  for  U.S.  Department  of  Defense 
Modeling  and  Simulation 

Ivar  Oswalt,  Tim  Cooley,  William  Waite,  Elliot  Waite,  Steve  “Flash”  Gordon, 
Richard  Severinghaus,  Jerry  Feinberg,  and  Gary  Lightner 


Employing  Risk  Management  to  Control  Military  Construction  Costs 

LTC  Steven  M.  F.  Stuban,  USA  (Ret.),  Thomas  A.  Mazzuchi,  and 
Shahram  Sarkani 


Mitigating  Spirals  of  Conflict  in  DoD 
Source  Selections 

Steven  M.  Maser  and  Fred  Thompson 


Maximizing  Federal  IT  Dollars:  A  Connection 
Between  IT  Investments  and  Organizational 
Performance 

BG  Ennis  C.  “Jim”  Whitehead,  USAR  (Ret.), 
Shahram  Sarkani,  and  Thomas  A.  Mazzuchi 


April  2011  Vol.  18  No.  2 


ISSUE  58 


2AU 


Defense  Acquisition  University 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

APR  2011  2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2011  to  00-00-2011 

4.  TITLE  AND  SUBTITLE 

Defense  AR  Journal.  Volume  18,  Number  2,  Issue  58.  April  2011 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Defense  Acquisition  University  (DAU), Defense  Acquisition  Research 
Journal, 9820  Belvoir  Road,  Suite  3, Fort  Belvoir,VA, 22060-5565 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  US 

unclassified  unclassified  unclassified  Report  (SAR) 

126 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Defense  Acquisition  University 


Katharina  G.  McFarland 

President,  Defense  Acquisition  University 


Linda  B.  Oliver 

Office  of  Small  Business  Programs, 
Department  of  Defense 

Shay  Assad 

Director,  Defense  Procurement 
and  Acquisition  Policy,  Office  of  the 
Under  Secretary  of  Defense  ( Acquisition , 
Technology  and  Logistics) 

Dr.  J.  Ronald  Fox 

Professor  Business  Administration, 
Emeritus  Harvard  Business  School 

ADM  James  R.  Hogg,  USN  (Ret.) 

Director,  Chief  of  Naval  Operations, 
Strategic  Studies  Group 


Dr.  Richard  Donnelly 

The  George  Washington  University 

Tim  Shannon 

Defense  Acquisition  University 

Dr.  Ned  Kock 

Texas  A&M  International  University 

Dr.  Christopher  G.  Pernin 

RAND  Corporation 

Dr.  Keith  Snider 

Naval  Postgraduate  School 

Dr.  Kevin  Buck 

MITRE  Corporation 

Dr.  Michelle  Bailey 

Defense  Acquisition  University 


Board  of  Review 

Nancy  Heimbaugh 

Director,  Acquisition  Management, 
Senior  Procurement  Executive,  and 
Component  Acquisition  Executive, 
Defense  Logistics  Agency 

Dr.  Ashton  B.  Carter 

Under  Secretary  of  Defense 
(Acquisition,  Technology  and  Logistics) 

Dr.  Diane  R.  Murphy 

Chief  Executive  Officer/Founder 
Information  Technology  Management 
Institute 

Editorial  Board 

Dr.  Larrie  D.  Ferreiro 

Chairman  and  Executive  Editor 

Dr.  Roy  Wood 

Defense  Acquisition  University 

Mike  Kotzian 

Defense  Acquisition  University 

Robert  Skertic 

Defense  Acquisition  University 

Dr.  Don  McKeon 

Defense  Acquisition  University 

Dr.  John  Snoderly 

Defense  Acquisition  University 

Dr.  Craig  Lush 

Defense  Acquisition  University 

Dr.  Mark  Montroll 

Industrial  College  of  the  Armed  Forces 


Patrick  J.  Fitzgerald 

Director, 

Defense  Contract  Audit  Agency 

Eleanor  Spector 

Wee  President  of  Contracts, 

Lockheed  Martin  Corporation 

Kathryn  C.  Turner 

President  and  Chief  Executive  Officer, 
Standard  Technology,  Inc. 


Dr.  Richard  Shipe 

Industrial  College  of  the  Armed  Forces 

Richard  Altieri 

Industrial  College  of  the  Armed  Forces 

RADM  James  Greene,  USN  (Ret.) 

Naval  Postgraduate  School 

Dr.  Don  Birchler 

Center  for  Naval  Analyses  Corporation 


ISSN  2156-8391  (print)  ISSN  2156-8405  (online) 

The  Defense  Acquisition  Research  Journal,  formerly  the  Defense  Acquisition  Review  Journal,  is  published  quarterly 
by  the  Defense  Acquisition  University  (DAU)  Press.  Postage  is  paid  at  the  U.S.  Postal  facility,  Fort  Belvoir,  VA,  and  at 
additional  U.S.  Postal  facilities.  Postmaster,  send  address  changes  to:  Editor,  Defense  Acquisition  Research  Journal, 
DAU  Press,  9820  Belvoir  Road,  Suite  3,  Fort  Belvoir,  VA  22060-5565.  For  free  copies,  mail  written  requests  with  an 
original  signature  to  the  above  address  using  the  subscription  form  provided  in  this  journal.  Some  photos  appearing 
in  this  publication  may  be  digitally  enhanced.  Articles  represent  the  views  of  the  authors  and  do  not  necessarily 
reflect  the  opinion  of  DAU  or  the  Department  of  Defense. 


\  we're  on  the  web  at  http://www.dau.mil/pubscats/Pages/ARI.aspx 


A  Publication  of  the  Defense  Acquisition  University 


The  Defense  Acquisition  Research  Journal  (ARJ)  is  a 

scholarly  peer-reviewed  journal  published  by  the 

Defense  Acquisition  University  (DAU).  All  submissions  receive 


Eduard  H.  Boyd 

Harambee  Dennis 

Collie  J.  Johnson 

Director  Visual  Arts  &  Press 

Technical  Editor 

Norene  L.  Fagan-Blanch 

Sydney  Coelho 

Managing  Editor 

Miracle  Riese 

Publications  Assistant 

Deputy  Director, 

o 

i 

T 

V. 

O 

c 

Visual  Arts  &  Press 

- iNuena  Ketsi - 

Nina  Costine 

Multimedia  Assistant 

Frances  Battle 

Senior  Graphic  Designer 

Production  Manager, 

Schatz  Publishing  Group 

Visual  Arts  &  Press 

Editing,  Design,  and  Layout 

122 


Calculating  Return  on  Investment  for  U.S.  Department  of  Defense 
Modeling  and  Simulation 

Ivar  Oswalt,  Tim  Cooley,  William  Waite,  Elliot  Waite, 

Steve  "Flash"  Gordon,  Richard  Severinghaus,  Jerry  Fein  berg, 
and  Gary  Lightner 


As  budgets  decrease,  it  becomes  increasingly  important  to  determine  the 
most  effective  ways  to  invest  in  modeling  and  simulation  (M&S).  This  article 
discusses  an  approach  to  comparing  different  M&S  investment  opportunities 
using  a  return  on  investment  (ROI)-like  measure.  The  authors  describe  methods 
to  evaluate  "benefit”  (i.e.,  increased  readiness,  more  effective  training,  etc.) 
received  from  an  investment  and  then  use  those  metrics  in  a  decision  analysis 
framework  to  evaluate  each  M&S  expenditure.  Finally,  they  conclude  by 
discussing  the  importance  of  viewing  M&S  investments  from  a  Department 
of  Defense  (DoD)  Enterprise  view,  evaluating  investment  over  multiple  years, 
measuring  well-structured  metrics,  and  using  those  metrics  in  a  systematic 
way  to  produce  an  ROI-like  result  that  DoD  can  use  to  evaluate  and  prioritize 
M&S  investments. 


Employing  Risk  Management  to  Control  Military  Construction  Costs 

LTC  Steven  M.  F.  Stuban,  USA  (Ret.),  Thomas  A.  Mazzuchi, 
and  Shah  ram  Sarkani 

Systems  acquisition  inherently  contains  elements  of  uncertainty  that  must  be 
effectively  managed  to  meet  project  cost,  schedule,  and  performance  objec¬ 
tives.  While  the  U.S.  Department  of  Defense  has  a  record  of  employing  systems 
engineering  technical  management  processes  (including  risk  management)  to 
address  these  uncertainties  for  major  weapon  systems  acquisition,  the  applica¬ 
tion  of  risk  management  to  Military  Construction  (MILCON)  projects  is  a  recent 
development.  This  research  studies  the  use  of  a  formal  risk  management  program 
on  a  MILCON  project  and  assesses  whether  such  use  influences  the  project’s 
total  cost  growth  relative  to  that  of  U.S.  Army  Corps  of  Engineers’  historical 
data.  A  case  study  methodology  is  employed  assessing  the  National  Geospatial- 
Intelligence  Agency  (NGA)’s  multibillion  dollar  NGA  Campus  East  program. 
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Mitigating  Spirals  of  Conflict  in  DoD  Source  Selections 

Steven  M.  Maser  and  Fred  Thompson 


Government  contracting  is  rife  with  opportunities  for  miscommunication 
and  misperception.  This  can  undermine  trust  and  fuel  spirals  of  conflict.  For 
this  article,  the  authors  interviewed  participants  and  analyzed  Government 
Accountability  Office  (GAO)  bid  protest  decisions  involving  Department  of 
Defense  source  selections.  They  found  agency,  vendor,  and  GAO  practices  that 
trigger  and  fuel  these  spirals.  Contracting  agencies  and  GAO  can  take  steps  to 
improve  communication,  reduce  inconsistencies,  and  reduce  perceptions  of  bias, 
thereby  mitigating  costly  bid  protests. 
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Maximizing  Federal  IT  Dollars:  A  Connection  Between  IT  Investments 
and  Organizational  Performance 

BG  Ennis  C.  “Jim"  Whitehead,  USAR  (Ret.),  Shahram  Sarkani, 
and  Thomas  A.  Mazzuchi 


Evaluating  how  best  to  invest  government  information  technology  (IT)  dollars 
means  making  choices.  Should  agencies  strengthen  infrastructure  with  energy- 
efficient  servers  and  increased  network  bandwidth,  purchase  software  to  cut 
costs,  increase  collaboration,  or  invest  more  to  meet  stakeholders’  future 
needs?  Is  there  a  connection  between  the  way  agencies  invest  IT  dollars  and 
successful  mission  accomplishment?  In  this  article,  the  authors  show  a  connec¬ 
tion  between  IT  investment  allocations  and  organizational  performance  in 
federal  government  agencies,  and  demonstrate  how  higher  performing  agen¬ 
cies  invest  differently  in  IT  than  lower  performing  agencies.  Federal  managers 
can  compare  their  organization’s  IT  investment  portfolio  with  high-performing 
agencies  and  compare  their  investment  allocations  with  other  federal  organi¬ 
zations  with  similar  missions  to  determine  optimum  IT  investment  allocations 
for  their  agencies. 
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From  the  Chairman  and  Executive  Editor 


FROM  THE  CHAIRMAN  AND 
EXECUTIVE  EDITOR 


In  this  issue,  I  am  pleased  to  spotlight 
the  winners  of  the  DAU  Alumni  Associa¬ 
tion’s  2011  Research  Paper  Competition. 

The  theme  for  this  year  is  “Making  Every 
Dollar  Count— Improving  Acquisition  Out¬ 
comes.”  The  topic  is  directly  relevant  to 
one  of  the  most  important  initiatives  in 
recent  years  to  come  from  Under  Secre¬ 
tary  of  Defense  for  Acquisition,  Technology  and  Logistics  Dr. 
Ashton  Carter:  the  mandate  to  deliver  better  value  to  the  tax¬ 
payer  and  warfighter  by  improving  the  way  the  Department 
of  Defense  does  business.  Dr.  Carter  specifically  tasked  the 
acquisition  community  to  search  for  means  to  achieve  pro¬ 
ductivity  growth;  in  other  words,  to  do  more  without  more.1 

The  winners  of  this  year’s  competition  have  made  sig¬ 
nificant  contributions  to  our  understanding  of  how  defense 
acquisition  can  become  more  effective  and  more  efficient. 
Ivar  Oswalt  and  his  colleagues  garnered  first  prize  with  their 
paper  “Calculating  Return  on  Investment  for  U.S.  Department 
of  Defense  Modeling  and  Simulation,”  which  describes  how 
modeling  and  simulation  expenditures  can  be  evaluated  from 
an  enterprise-wide  point  of  view.  Second  prize  is  awarded  to 
Steven  Stuban  and  his  co-authors  for  their  article  “Employ¬ 
ing  Risk  Management  to  Control  Military  Construction  Costs,” 
which  demonstrates  how  a  formal  risk  management  program 
was  effective  in  controlling  project  cost  growth. 

Two  papers  tied  for  third  prize.  Steven  Maser  and  Fred 
Thompson,  in  “Mitigating  Spirals  of  Conflict  in  DoD  Source 
Selections,”  explain  how  improved  communications  can 
mitigate  costly  bid  protests  that  often  plague  government 
contracts.  “Maximizing  Federal  IT  Dollars:  A  Connection 
between  IT  Investments  and  Organizational  Performance,”  by 
Jim  Whitehead  and  colleagues,  highlights  the  fact  that  orga¬ 
nizations  investing  more  in  IT  innovation  often  out-perform 
those  in  which  innovation  plays  a  lesser  role  in  their  invest¬ 
ment  portfolio. 

Many  other  authors  contributed  noteworthy  articles  to  this 
competition.  However,  space  permits  us  to  publish  only  one 
of  those  contributions  in  this  issue:  “Moving  Toward  Improved 
Acquisition  Outcomes,”  by  Everett  Roper,  which  investigates 
the  links  between  organizational  culture  and  acquisition  out- 


comes.  To  all  of  the  contributors,  we  give  our  heartfelt  thanks 
for  their  efforts.  We  also  thank  those  who  reviewed  the  multi¬ 
tude  of  articles  we  received,  and  took  on  the  task  of  selecting 
winning  articles  from  a  pantheon  of  excellent  submissions. 

In  this  issue,  we  add  to  the  Defense  Acquisition  Profes¬ 
sional  Reading  List  with  a  review  by  Shannon  Brown  (National 
Defense  University)  of  Building  the  Trident  Network  by  Maggie 
Mort— an  examination  of  how  organizations,  technologies,  and 
communities  all  converged  to  create  the  United  Kingdom’s 
Trident  submarine  and  missile  system. 

On  a  final  note,  I’m  pleased  to  note  that  DAU  has  wel¬ 
comed  aboard  its  new  President,  Katharina  McFarland,  who 
brings  to  the  position  a  wealth  of  knowledge  and  experience 
in  defense  acquisition. 


< 


Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 


ENDNOTE 


1.  Carter,  A.  B.  (2010).  Better  buying  power:  Guidance  for  obtaining  greater  efficiency 
and  productivity  in  defense  spending  [Memorandum].  Washington,  DC:  Office  of  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics. 
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ON  INVESTMENT  FOR 
U.S.  DEPARTMENT  OF 
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1 Ivar  Oswalt,  Tim  Cooley,  William  Waite, 
Elliot  Waite,  Steve  "Flash"  Gordon, 
Richard  Severinghaus,  Jerry  Fein  berg, 
and  Gary  Lightner 

As  budgets  decrease,  it  becomes  increasingly  important 
to  determine  the  most  effective  ways  to  invest  in  modeling 
and  simulation  (M&S).  This  article  discusses  an  approach  to 
comparing  different  M&S  investment  opportunities  using  a 
return  on  investment  (ROI)-like  measure.  The  authors  describe 
methods  to  evaluate  “benefit”  (i.e.,  increased  readiness,  more 
effective  training,  etc.)  received  from  an  investment  and  then 
use  those  metrics  in  a  decision  analysis  framework  to  evaluate 
each  M&S  expenditure.  Finally,  they  conclude  by  discussing  the 
importance  of  viewing  M&S  investments  from  a  Department 
of  Defense  (DoD)  Enterprise  view,  evaluating  investment  over 
multiple  years,  measuring  well-structured  metrics,  and  using 
those  metrics  in  a  systematic  way  to  produce  an  ROI-like  result 
that  DoD  can  use  to  evaluate  and  prioritize  M&S  investments. 


Keywords:  Modeling  and  Simulation  (M&S), 
Investment,  Return  on  Investment  (ROI),  Efficiency, 
Value 
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Successful  Department  of  Defense  (DoD)  Enterprise  modeling 
and  simulation  (M&S)  investment  requires  structure,  persistence, 
and  common  valuation  for  effective  execution.  The  methodology 
summarized  in  this  article  provides  a  systematic  process,  based 
upon  theoretical  aspects  of  capital  structure,  by  which  DoD  invest¬ 
ments  in  M&S  can  be  compared,  evaluated,  and  directed  to  achieve 
the  greatest  return  on  investment  (ROI)  in  this  “national  critical 
technology”  (House  Resolution  [H.  Res.]  487,  2007). 

To  effectively  apply  a  technology  like  M&S  to  a  DoD  Enterprise, 
application,  or  program,  it  is  critical  to  define  and  assess  rigorous 
measures  of  merit  and  metrics  that  reflect  the  results  of  M&S  appli¬ 
cation  across  the  relevant  spectra  of  management,  mission,  and 
system.  Such  assessments  are  especially  critical  as  budgets  are 
reduced,  opportunities  for  live  tests  and  exercises  are  curtailed, 
and  acquisition  time  lines  are  shortened.  Currently,  most  M&S  value 
assessments  use  metrics  that  are  uneven  in  scope,  very  case-spe¬ 
cific,  do  not  allow  consistent  aggregation,  or  are  not  well  structured. 
Additionally,  some  measures  that  are  used,  like  ROI,  are  actually 
incorrectly  defined;  others,  however,  are  undefined,  thus  making  the 
assertions  of  value  at  best  vague,  and  at  worst  incorrect.  Finally,  all 
too  often  important  distinctions  are  not  made  between  and  among 
terms  critical  to  consistent  ROI  assessment,  such  as  metrics,  mea¬ 
sure,  scale,  quantity,  quality,  cost,  utility,  and  value. 

Prior  efforts  to  characterize  the  cost-benefits  of  M&S  have 
included  surveys,  assessments,  and  methodological  developments. 
Surveys  summarize  the  results  of  efforts  already  conducted  (Wor¬ 
ley,  Simpson,  Moses,  Aylward,  Bailey,  &  Fish,  1996).  Methodological 
development  articles  provide  insights  into  how  to  improve  M&S 
value  calculation  (Gordon,  2006).  Assessments  typically  provide 
insights  based  on  one  of  four  approaches:  nominal  description, 
case-based,  business-oriented,  or  multi-attribute  examination. 
All  four  have  advanced  the  state-of-the-art  in  M&S  assessment, 
but  have  not  yielded  an  overall,  rigorous,  and  effective  approach 
for  placing  metrics  in  a  decision  analysis  framework  to  allow  the 
evaluation  of  M&S  investment.  The  methodology  developed  here 
(Figure  1)  is  distinctive  insofar  as  it  provides  prescriptive  guidance 
while  allowing  for  the  comparison  of  alternative  M&S  investments 
(M&S  compared  to  other  M&S  or  M&S  compared  to  other  alterna¬ 
tives  [analysis,  war  games,  etc.])  to  support  a  mission  or  meet  a  goal. 

It  also  facilitates  an  assessment  of  an  M&S  alternative  over  time 
(how  the  capabilities  provided  change  from  the  initial  application  to 
subsequent  use).  Such  time-considerate  assessments  are  especially 
critical  in  today’s  environment  of  shrinking  budgets.  By  viewing 
investments  from  a  DoD  Enterprise  view,  evaluating  investment  over 
a  multiyear  time  line,  measuring  metrics  developed  from  this  view- 
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FIGURE  1.  M&S  INVESTMENT  METHODOLOGY 
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point,  and  using  these  metrics  in  a  systematic  way  to  produce  an 
ROI-like  result,  the  DoD  can  evaluate  and  prioritize  M&S  investment. 

Market  Context  and  Business  Practice 

Stand-alone  strategies  don’t  work  when  your  company’s  suc¬ 
cess  depends  on  the  collective  health  of  the  organizations 
that  influence  the  creation  and  delivery  of  your  product. 
Knowing  what  to  do  requires  understanding  the  ecosystem 
and  your  organization’s  role  in  it.  (lansiti  &  Levien,  2004) 

This  quote  from  the  Harvard  Business  Review  addresses  the 
fundamental  premise  that  commercial  businesses  exist  and  thrive 
(or  not)  within  the  context  of  a  business  environment  much  larger 
than  exists  within  the  boundaries  of  an  individual  firm.  To  succeed, 
individual  firms  must  learn  to  recognize  and  create  value  within 
“the  ecosystem”  in  which  they  exist.  Translated  to  the  domain  of 
DoD  M&S  Enterprise  management,  the  quote,  as  interpreted  by  the 
authors,  could  read: 

Stand-alone  M&S  strategies  don’t  work  when  DoD’s  suc¬ 
cess  depends  on  the  collective  value  created  across  the 
Enterprise,  and  its  creation  and  delivery  of  value  derived 
from  its  investment  in  M&S.  Knowing  what  to  do  requires 
understanding  DoD’s  ecosystem  and  leadership’s  role  in  it. 

Within  the  DoD,  many  organizations  influence  the  creation  of 
value  from  M&S  investment.  On  an  Enterprise  level,  the  key  to  maxi¬ 
mizing  value  is  understanding  who  shoulders  the  costs  and  who 
potentially  derives  value  from  the  allocation  of  resources  to  M&S. 
DoD  investment  strategies  need  to  address,  at  a  minimum,  these 
aspects  of  economic  valuation: 

•  Government  (DoD)  being  the  (only)  buyer  in  many  parts 
of  its  M&S  market  does  discriminate  it  from  private- 
sector  M&S  investment. 
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•  A  lack  of  “marketplace”  from  which  to  gauge  economic 
valuation  often  complicates  DoD’s  efforts  to  make 
sound,  credible  valuation  judgments. 

•  Government  must  account  for  intangible  benefits  as 
contrasted  to  monetized  benefits  or  simple  revenue. 

•  Unlike  commercial  practice  (e.g.,  corporation-  or  com¬ 
pany-based),  when  the  DoD  invests,  a  misalignment 
often  occurs  between  the  “cost  bearer”  (the  resource 
sponsor)  and  the  “benefit  accruer”  (the  group  that 
gains  an  advantage  from  the  investment),  especially 
when  the  investment  creates  and  returns  value  to  DoD 
components  that  exceeds  the  expected  ROI. 

The  last  bullet  is  particularly  significant.  In  assessing  a  candi¬ 
date  investment,  a  practice  or  methodology  does  not  exist  in  the 
DoD  to  capture  and  characterize  the  future  and  extended  value 
accruing  to  users  beyond  the  primary  recipients  of  the  investment. 
Having  a  methodology  to  capture  such  extended  benefits  could 
change  the  outcome  of  an  investment  decision  from  “not  possible” 
to  “approved,”  and  provide  a  mechanism  for  assessing  all  benefi¬ 
ciaries  for  their  fair  share  of  investment  costs.  Additional  difficulties 
arise  in  the  fact  that  in  many  cases  the  DoD  M&S  investment  cannot 
be  monetized  (translating  elements  of  value  to  units  of  dollars)  in 
a  manner  analogous  to  commercial  business.  Placing  a  monetary 
amount  on  lives  saved,  readiness  improved,  or  warfighters  better 
trained  is  difficult  if  not  impossible.  The  DoD’s  characterization  of 
value  must  often  be  in  terms  that  are  naturally  qualitative,  making 
the  calculation  of  extended  benefit  (analogous  to  the  time-value- 
of-money)  very  different  than  in  the  commercial  sector. 

Across  the  DoD,  the  present  practice  is  to  base  investment  in 
M&S  on  a  number  of  methods;  at  an  Enterprise  level,  however,  the 
practice  is  neither  systematic  nor  consistent.  Writing  in  Acquisition 
Review  Quarterly,  the  Army  Developmental  Test  Command  Director 
for  Test  and  Technology  C.  David  Brown  and  co-authors  G.  Grant, 
D.  Kotchman,  R.  Reyenga,  and  T.  Szanto  wrote: 

Most  program  managers  justified  their  M&S  investment 
based  on  one  or  more  of  the  following:  reducing  design  cycle 
time;  augmenting  or  replacing  physical  tests;  helping  resolve 
limitations  of  funds,  assets,  or  schedules;  or  providing  insight 
into  issues  that  were  impossible  or  impracticable  to  examine 
in  other  ways.  (Brown  et  al.,  2000) 

Simply  put,  program  managers  (PMs)  are  under  intense  pressure 
to  complete  their  programs  on  budget  and  within  time  lines.  They 
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lack  an  institutional  mandate  to  develop  or  use  M&S  tools  that  may 
have  wider  application  to  other  programs,  or  that  will  be  cheaper 
to  operate  and  sustain  in  the  long  term  (Brown  et  al.,  2000).  This 
focus  on  the  program  level,  while  potentially  good  for  the  PMs, 
can  be  detrimental  to  the  Enterprise  at  large.  When  considering  an 
allocation  of  resources,  PMs  must  consider  not  only  costs,  but  also 
explicitly  definable  benefits.  Equally  important  at  the  Enterprise 
level  are  values  (economies  of  scope),  which  must  be  assigned  by 
leadership  to  complete  the  process  of  estimating  ROI  and  other 
measures  of  value  with  respect  to  M&S  assets.  The  methodology 
proposed  here  is  a  step  in  accounting  for  these  competing,  yet 
equally  important  value  metrics. 

Stakeholder  and  Community  of 
Practice  Specification 

Understanding  stakeholders  and  their  role-dependent  sensi¬ 
tivities  within  the  M&S  community  of  practice  provides  the  context 
within  which  to  determine  M&S  metrics.  DoD  stakeholders  oper¬ 
ate  within  a  broad  M&S  market,  where  “market”  includes  the  full 
economic  landscape  over  which  M&S  products  and  services  have 
impact.  DoD  M&S  stakeholders  fall  into  seven  categories: 

1.  Consumers/Users— End  users  of  M&S-powered  prod¬ 
ucts  or  of  M&S  services 

2.  Buyers— Expenders  of  funds  for  M&S-powered  products 
or  of  M&S  services 

3.  Sellers— Providers  of  M&S  tools,  data,  or  services 

4.  Investors— Providers/appropriators/deciders  on  expen¬ 
ditures  of  funds  for  M&S  products  or  services 

5.  Approvers/Raters— Providers  of  a  “seal  of  approval”  for 
M&S  tools,  data,  or  services 

6.  Reviewers— Providers  of  “advice  and  consent”  on  M&S 
issues,  including  M&S  products  or  services 

7.  Promoters/Advocates— Independent  providers  of 
“encouragement”  to  the  development  of  the  M&S  mar¬ 
ket  for  M&S-powered  products  or  services 

Each  stakeholder  category  comes  to  the  M&S  market  with  a 
role-dependent  perspective.  These  perspectives  are  designated 
as:  Program,  Community,  Enterprise,  Federal,  and/or  Society.  For 
DoD  M&S  investment,  the  first  three  perspectives— Program,  Com¬ 
munity,  and  Enterprise— are  considered  to  be  internal  to  the  DoD. 
The  final  two— Federal  and  Society— are  considered  to  be  external 
to  the  DoD.  Stakeholders  provide  another  dimension  that  is  useful 
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in  characterizing  DoD  M&S  investment  considerations  and  elements 
of  value. 

1.  Program  stakeholders’  concerns  focus  on  applicability, 
availability,  and  affordability;  credibility,  analytic  sound¬ 
ness,  user  friendliness,  and  entertainment  delivered, 
as  well  as  modularity,  interoperability,  and  portability; 
and  concentrate  on  systems-of-systems  or  system-level 
functionality. 

2.  Community  stakeholders’  focus  is  on  managing  M&S 
within  specific  areas  such  as  acquisition,  analysis,  plan¬ 
ning,  testing,  training,  and  experimentation,  and  is 
oriented  toward  application-level  indicators  of  success 
or  failure. 

3.  Enterprise  stakeholders’  concerns  focus  on  M&S  capabil¬ 
ities  that  apply  across  diverse  activities  of  the  Services, 
combatant  commands,  and  DoD  agencies. 

4.  Federal  stakeholders’  concerns  focus  on  M&S  develop¬ 
ments  across  departments  and  agencies  of  the  U.S. 
Government. 

5.  Society  stakeholders’  concerns  focus  on  the  role  and 
impact  of  M&S  on  governments,  cultures,  academia, 
industries,  and  populations. 

These  concerns  are  broad  and  encompassing,  and  include  stan¬ 
dards,  policies,  management,  tools,  and  people,  along  with  reuse, 
interoperability,  collaboration,  interactiveness,  and  sharing  of  assets 
in  a  defense-wide  manner. 


Use  Case 

Developing  and  understanding  use  cases,  including  stakeholder 
needs  and  requirements,  help  determine,  refine,  and  evaluate  the 
process  for  defining  M&S  investment  metrics.  Use  cases  illustrate 
stakeholder  issues  and  role-dependent  sensitivities  together  with 
investment  decision  processes,  and  serve  to  support  and  guide 
the  definition,  explanation,  and  evaluation  of  processes  and  metric 
alternatives.  We  have  developed  a  framework  that  encompasses  a 
consistent  and  complete  set  of  use-case  descriptions  for  use  in  the 
analysis  of  M&S  investment  metrics.  Table  1  lists  the  parameters  of  a 
framework  that  provides  a  consistent  and  complete  set  of  use-case 
descriptions  to  help  analyze  M&S  investment  metrics.  The  full  report 
of  the  study  details  three  use  cases  from  different  perspectives 
(AEgis  Technologies  Group,  2008).  The  use  cases  examine  exercise 
options,  Live,  Virtual,  Constructive  (LVC)  middleware  choices,  as 
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TABLE  1.  M&S  USE  CASE  FRAMEWORK 


Parameter 

Selected  Values 

What/Where 

Investment  situation,  investment  goal,  investment 
time  line,  asset  types,  asset  numbers,  other  asset 
information,  geographical  constraints 

Who 

Stakeholder  market  category,  stakeholder 
perspective,  stakeholder  office 

Why 

Concerns,  issues,  forcers,  drivers,  constraints 

When 

Near-term  investments,  mid-term  investments, 
long-term  investments,  schedule  constraints 

How 

Costs  (near  term,  mid  term,  long  term) 

So  What 

Result,  benefit,  utility,  cost  savings 

Data  Support 

Sources,  pedigree,  availability,  timeliness 

well  as  conceptual  modeling  alternatives  for  the  Missile  Defense 
Agency.  In  each  of  these,  the  steps  in  this  process  are  delineated 
and  discussed,  sample  data  included,  and  a  decision  recommended 
based  upon  the  given  scenario.  Due  to  space  limitations,  we  were 
unable  to  include  them  in  this  article. 

Assets 

To  fully  understand  DoD  M&S  investment,  it  is  also  critical  to 
identify  those  items  that  DoD  buys.  We  first  define  the  difference 
between  assets  (items  for  DoD  investment)  and  consumables  (in 
accounting  terms:  expenses).  Then  we  list  the  assets  and  catego¬ 
rize  them  depending  upon  the  point  of  view  (POV).  For  example, 
if  one  views  assets  from  the  DoD  POV  (Acquisition,  Analysis,  Plan¬ 
ning,  Training,  Experimentation,  and  Testing),  then  the  assets  are 
categorized  one  way.  Alternatively,  if  the  POV  is  that  of  the  DoD 
Enterprise  Community  (which  from  its  M&S  Vision  Statement  articu¬ 
lates  the  categories  of  Infrastructure,  Policies,  Management,  Tools, 
and  People),  then  the  assets  are  characterized  differently  (AEgis 
Technologies  Group,  2008). 

From  a  DoD  perspective,  an  asset  is  defined  as:  "Something  of 
monetary  value,  owned  by  DoD,  that  has  future  benefit."  A  consum¬ 
able,  on  the  other  hand,  is:  "Something  capable  of  being  consumed; 
that  may  be  destroyed,  dissipated,  wasted,  or  spent.”  The  primary 
difference  is  the  concept  of  future  benefit.  “Future”  in  this  sense  is 
typically  thought  of  as  more  than  12  months  in  the  future.  Exam¬ 
ples  of  DoD  M&S  assets  include:  F-16  simulators,  the  Navy’s  Battle 
Stations  21  simulator,  and  the  online  game  “America’s  Army.”  Con¬ 
sumables,  on  the  other  hand,  are  items  such  as  paper,  pencils,  jet 
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TABLE  2.  ASSETS  LISTING 


Hardware 

Software 

Networks 

Facilities 

People 

Products  & 
Procedures 

Computers 

Models 

Communication 

Lines 

Buildings 

Expertise 

Plans/Policies 

Electronic 

Hardware 

Simulation 

Architecture 

Labs 

Experience 

Standards 

Hardware  in 

the  Loop 

Tools 

(CAD/CAM)* 

Transaction 

Protocols 

Ranges 

Skills/ 

Education 

Analysis  Results 

Mock-ups 

Data/ 

Databases 

Physical 

Models 

Operational 

Knowledge 

Conceptual 

Models 

Spares 

Repositories 

Management 

Processes 

* Computer-Aided  Design/Computer-Aided  Manufacturing 


fuel,  printer  ink— all  typically  used  and  depleted  within  12  months  of 
purchase.  In  light  of  this,  those  types  of  items  that  constitute  DoD 
investment  assets,  using  the  DoD  M&S  Vision  Statement  and  the 
DoD  Communities,  are  shown  in  Table  2. 

By  comparing  this  categorization  with  that  developed  by  cross¬ 
mapping  this  list  with  the  DoD  Communities  (from  both  mission 
and  organizational  views),  and  with  the  DoD  M&S  Vision  Statement 
categories,  we  noted  some  interesting  relationships.  To  start,  every 
listed  asset  correlates  to  more  than  one  major  DoD  Community.  For 
example,  every  DoD  M&S  Community  invests  in  the  Asset  Models. 
While  this  is  not  surprising,  it  shows  that  there  may  be  efficiencies 
gained  by  studying  the  Enterprise  view  and  how  the  DoD  invests 
in  models  since  that  investment  is  widespread.  Also,  the  assets  are 
quite  varied  from  the  tangible  items  to  the  esoteric.  This  means  that 
some  assets  are  easy  to  value,  making  the  determination  of  the  cost 
of  the  investment  relatively  straightforward,  and  some  extremely 
difficult.  Finally,  it  is  difficult  to  place  assets  neatly  into  bins.  All 
assets  cross  functional,  mission,  organizational,  M&S  Community, 
and  DoD  M&S  vision  category  lines,  meaning  an  investment  in  any 
one  of  these  assets  affects  multiple  commands,  agencies,  and  per¬ 
haps  Services.  All  categories  and  sub-categories  invest  in  multiple 
assets.  Because  of  this,  to  be  the  most  effective  and  get  the  highest 
ROI,  investing  in  M&S  needs  to  be  viewed  at  the  Enterprise  level, 
not  at  an  individual  Community  level.  A  true  measure  of  investment 
effectiveness  cannot  be  achieved  unless  one  considers  all  the  costs 
and  benefits. 
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Asset  Costs 

A  decision  to  purchase  or  modify  an  M&S  asset  should  be  based 
on  the  needs  of  the  customer(s)  and  the  cost  of  the  purchase  or 
modification.  That  cost  and  associated  decisions  are  best  under¬ 
stood  within  the  context  of  multiyear  fiscal  calculations.  In  looking 
at  costs  and  the  ROI  of  those  costs,  it  is  important  to  again  acknowl¬ 
edge  that  business  and  government  operate  differently.  If  a  business 
were  to  purchase  an  asset,  the  business  owner  would  likely  evaluate 
the  impact  of  the  asset  on  the  bottom  line:  profit.  The  owner  would 
likely  predict  the  changes  in  profit  and  the  costs  to  purchase  or 
modify  the  asset  over  the  useful  lifetime  of  the  asset,  and  then  com¬ 
pute  (“discount”)  all  those  changes  in  profit  and  asset  costs  back 
to  the  current  year  (today’s)  decision  point.  Different  options,  such 
as  “purchase  asset  A”  or  “modify  asset  B,”  can  be  compared  in  this 
way,  even  if  these  options  have  different  payoff  and  cost  streams 
over  varying  numbers  of  years.  The  comparison  of  the  options  in 
terms  of  current-year  dollars  at  the  time  of  the  decision  gives  a 
standard  metric  that  allows  a  fair  evaluation  of  the  alternatives. 

Government  and  industry  cost  comparisons  differ  in  that  while 
government  generally  does  not  compute  profit,  it  does  compute 
changes  in  expenses.  Additionally,  in  government  the  changes  one 
stakeholder  or  one  PM  makes  can  have  cost  impacts  on  another 
PM,  so  one  PM  can  show  cost  savings  while  others  have  the  burden 
of  increased  costs  because  of  a  change  in  an  asset.  This  shows 
once  again  that  considering  the  Enterprise  perspective  across  all 
impacted  programs  is  essential  to  calculating  an  accurate  and  com¬ 
plete  value  of  M&S  investment. 

Typically,  cost  elements  for  M&S  assets  can  be  grouped  into 
useful  classifications  (Office  of  the  Director,  2007)  for  evaluation  of 
alternatives  through  the  calculation  of  current-year  metrics: 

•  Infrastructure:  standards,  architectures,  networks,  and 
environments 

•  Policies  at  the  Enterprise  level  (including  interoper¬ 
ability  and  reuse) 

•  Management  processes  for  models,  simulations,  and 
data 

•  Tools  in  the  form  of  models,  simulations,  and  authori¬ 
tative  data 

•  People  (including  well-trained  and  experienced  users) 


The  overall  study  illustrated  how  an  increased  level  of  granular¬ 
ity  for  these  classifications  could  be  tailored  to  the  project  and  asset 
particulars,  and  could  be  used  to  facilitate  the  calculation  of  costs 
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by  year  (AEgis  Technologies  Group,  2008).  The  following  example 
illustrates  the  type  of  M&S  alternatives  that  could  be  evaluated  using 
a  cost  element  structure  to  characterize  costs  of  several  alternatives 
over  several  years. 


Using  the  Cost  Element  Structure  to 
Compare  Alternative  M&S  Courses  of  Action 

A  simulation  professional  was  directed  to 
establish  an  annual  experiment  in  Alaska  to  evalu¬ 
ate  capabilities  such  as  the  combat  benefits  of  a 
new  system  for  position  determination  of  friendly 
ground  forces.  The  simulationist  will  need  to  evaluate 
alternative  simulations  for  use  in  this  annual  experi¬ 
ment.  Could  a  different  simulation  be  used  each 
year  depending  on  what  systems  are  being  evalu¬ 
ated,  or  would  it  be  acceptable  and  cheaper  to  use  a 
standard  core  simulation  over  the  next  5  years?  The 
cadre  of  simulation  operators  is  limited  in  Alaska, 
so  the  simulationist  must  also  evaluate  distribution 
of  the  simulation  environment  from  other  locations. 

In  this  first  year,  the  position  determination 
system  may  need  to  be  simulated  or  assumed.  Data¬ 
bases  for  geography  and  other  environmental  factors 
may  need  to  be  purchased  with  requisite  lead  time. 
Connectivity  and  simulation  architecture  costs  will 
have  to  be  evaluated.  The  estimated  cost  of  conduct¬ 
ing  the  experiment,  using  all  live  forces,  would  be  the 
most  costly  option,  and  could  be  used  to  estimate 
cost  avoidance  for  the  other  LVC  options. 

Depending  on  the  alternatives  evaluated,  some 
may  be  more  costly  in  the  current  year  and  cheaper 
in  the  out-years;  while  others  may  be  cheaper  in 
the  current  year  but  with  a  high  stream  of  out-year. 
Hence,  the  cost  comparison  of  the  alternatives  is 
evaluated  based  on  the  sum  of  the  discounted  costs 
across  the  entire  5  years  of  the  experiment. 


Results 
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need  to  be  rigorously  described  in  a  manner  that  accounts  for  both 
qualitative  and  monetary  dimensions.  The  approach  developed 
and  detailed  in  this  section  describes  the  metrics  required  for  such 
analyses,  including  types,  variability,  and  application  particularities. 
The  development  of  such  metrics  is  especially  important  in  M&S, 
where  the  impact  of  investment  and  application  is  not  exclusively 
monetary,  naturally  quantitative,  or  sometimes  even  intuitively  obvi¬ 
ous.  Where  the  word  “results”  appears,  its  use  reflects  the  outcome 
of  M&S;  includes  both  positive  and  negative;  encompasses  terms  like 
value,  utility,  contribution,  benefit,  and  return;  and  allows  for  both 
monetary  and  qualitative  effects. 


The  results  calculation  methodology  begins  with  a  series  of 
assumptions  and  definitions.  It  is  assumed  that  decision  makers  in  a 
governmental  agency  are  rational  actors  who  seek  to  optimize  rel¬ 
evant  outcomes.  Also,  outcomes  can  be  characterized  using  terms 
that  reflect  the  investment  value  of  alternatives  (meaning,  no  hid¬ 
den  agendas  or  overriding  private  concerns).  The  next  assumption 
is  that  the  metrics  can  be  accurately  quantified  (whether  inherently 
numeric  [like  money]  or  subjectively  assessed).  For  this  effort,  we 
define  three  organizing  principles  or  perspectives  that  can  be  con¬ 
sistently  applied:  Program,  Community,  and  Enterprise.  Next,  it  is 
important  to  understand  the  scope  of  the  results  determination. 
For  instance,  will  they  be  used  to  compare  alternatives  in  meeting 
a  goal  (M&S  to  M&S  or  M&S  to  other  options),  or  to  the  evolution  of 
an  M&S  capability  over  time?  Next,  in  calculating  results  metrics,  it  is 
important  to  define  the  term  “metric”  in  context  (Table  3).  The  next 
step  of  results  metric  calculation  is  measurement  or  assessment. 
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TABLE  3.  RESULTS  METRICS  CONTEXT 


Relationships 

Example 

First  are  the  classes/categories. 

e.g.,  Technical 

Associated  with  each  group  are 
characteristics/terms  describing  features. 

Maintainability,  Design 

Associate  these  with  more 

specific  properties. 

Mean  Time  Between 

Failures, Type 

Decompose  these  into  metrics,  standards 
of  measurement,  like  variables. 

Hours,  Days/Compiled, 
Interpreted 

Metrics  values  are  relative  to  a  scale  (a 
specified  graduated  reference  used  to 
measure)  and  may  be  nominal,  ordinal, 
interval,  or  ratio  in  type. 

1-2-3-4-5-6-7-8-9-10  /  C-l 

May  range  from  0  or  no  representation 
to  X,  which  X  represents  a  complete 
implementation  of  the  areas. 

Continuously  for  interval 

and  ratio  data 

Metrics  are  assigned  values,  based  on 
the  features  of  the  M&S  (the  act  of 
measurement)  or  mission  requirement. 

e.g.,  9,  Compiled 

Values  can  be  combined  into  aggregate 

measures  of  merit. 

C  =  2*1,  1  =  1,  Value  =  18 

The  focus  here  is  on  qualitative  or  subjective  judgments  that  can  be 
numerically  characterized  and  indices  that  are  naturally  quantitative. 
Finally,  it  is  often  very  important  to  aggregate,  calculate,  or  derive 
an  overall  measure  from  a  decision  theoretic  approach. 

Three  perspectives  apply  within  the  DoD  to  the  derivation  of 
relevant  M&S  results  metrics  and  the  calculation  of  their  ROI.  They 
are  the  Program  perspective,  which  includes  both  M&S  programs 
and  programs  or  activities  that  use  M&S  (Oswalt  &  Kasputis,  2006); 
the  Community  perspective,  as  described  in  the  Application  Area 
Descriptions  (Oswalt,  2005)  (i.e.,  the  “Surfboard  Chart”);  and  the 
Enterprise  perspective,  as  articulated  in  the  Strategic  Vision  for  DoD 
M&S.  Acknowledging  these  three  perspectives  is  critical,  since  the 
results  metrics  applicable  to  each  are  different  (Figure  2).  How- 
ever,  due  to  space  constraints  and  the  desirability  to  view  M&S 
investments  from  an  Enterprise  aspect,  only  Enterprise  metrics  are 
summarized  here. 

The  Enterprise  perspective  focuses  on  M&S  capabilities  that 
apply  “across  the  diverse  activities  of  the  Services,  combatant 
commands,  and  agencies”  and  thus  presents  goals  that  are  neces¬ 
sarily  broad  and  encompassing.  They  include  standards,  policies, 
management,  tools,  and  people  that  are  collaborative,  interactive, 
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FIGURE  2.  RESULTS  PERSPECTIVES 

_ Enterprise  (Brain) _ 

Leadership,  Implementation,  Business,  Infrastructure,  System  of  Systems 

Enterprise  metrics  reflect  orchestration  and  management-type 
activities 

Communities  (Organs) 

Design,  Manufacturing,  Sustainment,  Time  to  Market  Alternatives, 
Complexity,  Sensitivity,  Result  Time  Projection,  Familiarity, 
Comprehensive,  Decision  Time,  Test,  Design,  Augmentation, 
Extrapolation,  Completion  Time  Availability,  Scenario  Variation, 
Experimental,  Retention  Time  Discovery,  Doctrine,  Technology, 

Cycle  Time 

Community  metrics  reflect  more  specific  uses  and  yet  can  include  both 
enterprise-type  and  program-type  metrics  (when  the  program  crosses 
boundaries  within  a  community  or  between  communities) 

Programs  (Blood) _ 

Applicability,  Availability,  Affordability,  Rigorousness,  Engaging, 
Usability,  Creditability,  Technical 

Program  metrics  reflect  the  key  dimensions  of  individual  M&S  system 
developments  or  M&S  use  within  platform  development  of  programs 


and  sharing  of  assets  in  a  defense-wide  manner  that  includes  other 
“governmental  agencies,  international  partners,  industry,  and  aca¬ 
demia.”  Metric  categories  for  each  were  derived  previously  (Oswalt 
&  Tyler,  2008).  A  sample  set  of  Leadership  metrics  is  provided  in 
Table  4. 


ROI  Methods 

In  financial  analysis,  the  concept  of  return  is  critical  and  is  prin¬ 
cipally  used  to  measure  the  change  in  “value”  over  time.  As  such, 
return  is  used  by  the  Financial  Community  to  determine  two  impor¬ 
tant  concepts:  (a)  whether  or  not  the  benefit  of  an  investment  (or 
similar  action)  was  positive  or  negative— this  is  the  “direction”  of 
the  change;  and  (b)  how  positive  or  negative  the  change  was— this 
is  the  “magnitude.”  Financial  analysts  typically  calculate  only  one 
value  by  which  both  direction  and  magnitude  can  be  ascertained. 
The  use  of  a  single  value  is  possible  because  analysts  usually  com¬ 
pare  changes  in  a  single,  same  quantity:  U.S.  dollars.  The  two  most 
common  ways  to  measure  return  are  as  a  percentage  increase  in 
a  holding’s  value  between  two  time  periods.  Return  consists  of  (a) 
the  income  and  the  capital  gains  relative  to  an  investment.  It  is  usu- 
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TABLE  4.  ENTERPRISE  METRIC  SAMPLE 


Enterprise  Perspective 

Sample  Metrics 

Term 

(characteristics) 

Definition 

Quality 

Monetary 

Leadership  (class/category) 

Leadership 

Statement  of  vision 

and  associated 
advocacy/support 
of  timely  actions 

needed  for  an 

effective  enterprise 
(property) 

#/currency  of  vision 
and  resulting/ 
supporting  docs 
(metric) 

°  senior  leaders 

adopt  vision  within 
their  (other)  areas 

%  alignment  of 
funding  to  vision 
Savings  from 

reduced  unused  sunk 

costs 

Empowerment 

Developers, 

managers,  users 

that  are  engaged, 
asked,  able  to 
make  significant 

contributions 

#  innovative  ideas 

forwarded  without 

solicitation 

%  M&S  decision 

makers  attending 
key  meetings 

Reduction  in  costs 

to  get  new  M&S 
concepts 

Savings  from 

innovative  M&S  use 

Situational 

Awareness 

Decision  maker’s  and 

users'  understanding 

and  awareness  of 

M&S  standards,  tools, 

etc. 

#  meetings, 
conferences, 
repositories,  Web 
portals 

%  critical  information 

exchanged  among 

communities 

Reduction  in  costs 

to  finding  good  M&S 

information 

Cost  savings 

from  reduction  of 

duplicative  efforts 

Management 

Human  Capital 
Management  for 
recruiting,  assigning, 
career  development 

of  M&S  workforce 

%  M&S  billets  staffed 

with  M&S  qualified 
people 

%  M&S  qualified 
personnel  promoted/ 
retained 

Unnecessary 

training/retraining 

costs 

Cost-effective  M&S 

decisions 

Processes 

Adoption  of  rigorous, 
timely,  and  relevant 

standardization 

and  certification  of 

M&S  policy,  tools, 
workforce,  etc. 

#  promulgated 

processes 

consistently  adopted 
Decreased  product 
(policy,  tool,  etc.) 
generation  time 

Reduced  labor, 
travel,  and  software 

reworks 

Savings  from  error- 

rate  reduction 

ally  quoted  as  a  percentage  (INVESTOPEDIA®,  2010);  and  (b)  as  the 
amount  of  cash  (or  revenue)  generated  from  a  set,  fixed  asset  base, 
expressed  as  a  percentage  of  investment.  Examples  of  this  include 
Return  on  Equity,  Return  on  Assets,  Return  on  Common  Equity,  or 
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FIGURE  3.  DIAGRAM  OF  MADM  PROCESS  FOR  DoD  M&S 
INVESTMENT  ORGANIZED  BY  DoD  COMMUNITIES 


Input 

Input 

Input  | 

Input 

Input 

Input  | 

Input 

Input 

Input  | 

Input 

Input 

Input 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

ROI 

Metrics 

Other 

Metrics 

Acquisition 

Analysis 

Planning 

Training 

Experimentation 

People 

Return  on  Invested  Capital.  Both  of  these  methods  typically  use 
dollars  as  the  unit  of  measure. 

So  how  do  we  apply  the  concepts  of  financial  analysis  to  DoD 
M&S  projects?  The  concepts  of  magnitude  and  directionality  men¬ 
tioned  previously  are  essential  to  this  endeavor.  To  make  a  decision 
between  a  finite  set  of  options,  a  relative  sense  of  order  is  needed; 
that  is,  to  be  able  to  distinguish  which  project  is  better  than  the 
others.  Therefore,  while  we  might  not  assign  a  specific  dollar  value 
to  the  benefit  of  one  choice  over  another,  by  using  directionality 
and  magnitude,  we  can  arrive  at  a  “relative  ranking”  that  will  let  us 
compare  those  options  among  which  we  are  seeking  to  decide. 
Additionally,  the  notion  of  “internal  consistency”  in  evaluating  dif¬ 
ferent  options  is  vital.  If  we  are  not  able  to  gain  an  absolute  value 
(such  as,  say  83  percent),  but  are  to  rely  on  relative  values  (A  is 
better  than  B,  which  is  better  than  C),  we  must  make  sure  that  we 
are  consistently  applying  the  same  evaluation  criteria  to  all  the 
potential  choices.  The  methodology  for  evaluating  DoD  M&S  invest¬ 
ment  described  in  the  following  discussion  meets  these  criteria  and 
is  completely  consistent  with  the  manner  in  which  financial  analysis 
seeks  to  evaluate  return. 

Investment  Decision  Process 

Having  now  determined  metrics  for  the  costs  and  results  asso¬ 
ciated  with  an  investment,  we  are  in  a  position  to  decide  whether 
or  not  to  make  the  investment  using  these  metrics  and  others.  Our 
goal  is  to  employ  a  decision  process  that  takes  into  account  the  data 
gathered,  does  not  rely  upon  chance,  is  fundamentally  simple  to 
explain  and  defend,  and  is  consistent  (would  give  the  same  answer 
each  time  with  the  same  data). 
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Rational  actors,  when  faced  with  a  decision,  will  choose  the 
option  that  maximizes  their  gain  by  some  measure.  In  previous 
sections,  we  presented  methods  to  evaluate  the  costs  and  results 
of  an  M&S  investment;  noted  that  monetization  of  these  metrics 
may  be  difficult,  if  not  impossible,  to  perform  for  the  DoD;  and 
discussed  ROI  methods,  including  key  financial  analysis  elements. 
Given  this  environment  with  its  constraints,  we  developed  a  deci¬ 
sion  process  that  produces  an  ROI-like  quantitative  result  for  use 
in  M&S  investment  evaluation.  We  used  assessed  metrics  as  input 
to  a  Multi-Attribute  Decision  Making  (MADM)  network,  which  has 
the  qualities  of  being  robust,  relatively  explainable,  objective,  con¬ 
sistent,  and  once  established,  can  be  executed  fairly  simply.  MADM 
(Figure  3)  is  not  new  and  has  been  shown  to  work  well  in  structuring 
complex  decisions  involving  a  multidimensional  decision  space.  In 
its  simplest  form,  MADM  is  a  weighted  sum.  The  total  utility  score 
is  calculated  by  multiplying  each  attribute’s  normalized  input  score 
by  its  relative  weighting  (which  would  be  assigned  earlier)  and 
summing  all  the  products.  This  process  is  repeated  at  every  layer. 
While  other  formulae  can  be  employed  to  calculate  a  utility  score, 
the  weighted  linear  method  is  most  often  used  due  to  its  simplicity 
and  transparency  (Tompkins,  2003).  In  this  case,  multilayers  are 
desirable  for  a  few  reasons. 

First,  it  allows  for  the  higher  level  DoD  decision  makers  to  put 
different  emphasis  on  certain  communities  by  assigning  different 
weights  to  each  community.  Additionally,  multilayers  are  desirable 
for  transparency  since  grouping  the  metrics  by  community  makes  it 
easier  to  see  how  certain  measures  impact  the  overall  utility  score. 

It  should  be  noted  that  attributes  measured  should  be  mutually 
exclusive  (no  overlap)  to  prevent  one  attribute  from  influencing 
the  final  score  by  a  higher  amount  than  intended.  Additionally,  the 
weights  are  typically  set  by  a  team  of  subject  matter  experts,  which 
should  consist  of  experts  from  every  area  affected  by  the  deci¬ 
sion  under  consideration,  and  these  weights  should  be  reviewed 
regularly.  Finally,  risk  for  an  investment  can  be  incorporated  in  this 
process  either  as  its  own  category  or  as  a  cost  metric  input  to  the 
framework. 


Conclusions 

By  viewing  investments  from  a  DoD  Enterprise  perspective, 
evaluating  investment  over  a  multiyear  time  line,  measuring  metrics 
developed  from  this  POV,  and  using  these  metrics  in  a  systematic 
way  to  produce  an  ROI-like  result,  the  DoD  can  evaluate  and  pri¬ 
oritize  M&S  investment.  The  process  outlined  in  this  article  meets 
these  criteria  and  is  robust,  consistent,  and  adaptable.  If  followed, 
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the  prescribed  methods  and  guidelines  should  allow  the  DoD,  and 
similar  types  of  organizations,  to  make  M&S  investment  decisions 
that  result  in  an  increased  ROI  when  compared  to  the  current  state. 
An  important  next  step  in  the  development  and  use  of  this  meth¬ 
odology  is  its  application.  Whether  as  an  assessment  technique  for 
a  historical  examination  or  an  approach  for  future  M&S  investment 
analysis,  the  techniques  described  herein  would  provide  rigorous 
and  useful  insights. 
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Thomas  A.  Mazzuchi,  and  Shah  ram  Sarkani 

Systems  acquisition  inherently  contains  elements  of  uncer¬ 
tainty  that  must  be  effectively  managed  to  meet  project 
cost,  schedule,  and  performance  objectives.  While  the  U.S. 
Department  of  Defense  has  a  record  of  employing  systems 
engineering  technical  management  processes  (including 
risk  management)  to  address  these  uncertainties  for  major 
weapon  systems  acquisition,  the  application  of  risk  manage¬ 
ment  to  Military  Construction  (MILCON)  projects  is  a  recent 
development.  This  research  studies  the  use  of  a  formal  risk 
management  program  on  a  MILCON  project  and  assesses 
whether  such  use  influences  the  project’s  total  cost  growth 
relative  to  that  of  U.S.  Army  Corps  of  Engineers’  historical 
data.  A  case  study  methodology  is  employed  assessing  the 
National  Geospatial-Intelligence  Agency  (NGA)’s  multibillion 
dollar  NGA  Campus  East  program. 


Keywords:  Risk  Management,  Military  Construction 
(MILCON),  Construction,  U.S.  Army  Corps  of 
Engineers  (USACE),  National  Geospatial-Intelligence 
Agency  (NGA) 
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Risk  on  a  Military  Construction  (MILCON)  project  has  gener¬ 
ally  been  addressed  through  the  use  of  contingencies/reserves, 
specified  bonding  and  insurance  requirements,  and  inclusion  of 
appropriate  contract  clauses  at  the  onset  of  a  project  (Khadka  & 
Bolyard,  2010).  The  design,  construction,  and  commissioning  of  a 
facility  is,  however,  a  dynamic  process  engaging  numerous  par¬ 
ties.  Adhering  solely  to  relatively  static  measures  could  adversely 
constrain  the  project  team’s  ability  to  achieve  overarching  cost, 
schedule,  and  performance  objectives. 

While  DoD  has  provided  its  acquisition  professionals  ample 
guidance  on  the  need  for  implementing  risk  management  through¬ 
out  a  project’s  life  cycle,  it  does  so  in  the  context  of  major  weapon 
and  automated  information  systems  (Bolles,  2003).  As  was  noted  by 
a  former  Director  of  Military  Program,  U.S.  Army  Corps  of  Engineer 
Joe  Tyler,  this  guidance  has  only  recently  been  adapted  in  the  realm 
of  facility  acquisition  accomplished  through  MILCON  projects  (J.  J. 
Tyler,  personal  communication,  July  13,  2009). 

Additionally,  a  structured  approach  to  risk  management  from 
a  cost,  schedule,  and  performance  perspective  has  recently  been 
incorporated  into  the  Defense  Acquisition  University  (DAU)’s  Level 
III  certification  course  for  the  Facilities  Engineering  career  field 
(DAU,  2010). 

A  2008  survey  of  construction  industry  professionals  revealed 
that  respondents  are  managing  project  risks  roughly  61  percent 
of  the  time,  and  it  may  be  interpreted  to  mean  that  the  corner 
may  have  been  turned  regarding  use  of  formal  risk  management 
processes  (FMI  Corporation,  2008).  But  with  billions  of  dollars  com¬ 
mitted  annually  to  MILCON  projects,  one  must  ask  not  only  if  DoD’s 
current  level  of  formal  risk  management  processes  is  adequate,  but 
also  if  it  is  relevant. 

The  authors  of  this  research  used  a  case  study  format  in  assess¬ 
ing  the  application  of  risk  management  processes  on  the  National 
Geospatial-Intelligence  Agency  (NGA)  Campus  East  program.  In 
doing  so,  they  sought  to  define  the  process  that  was  employed  and 
to  assess  whether  it  was  effective  in  controlling  the  cost  growth  of 
the  facility  component  of  the  NGA  program. 

Background 

NGA  is  a  combat  support  agency  in  the  Department  of  Defense 
(DoD)  and  a  member  of  the  Intelligence  Community.  NGA’s  mis¬ 
sion  is  to  provide  geospatial  intelligence  in  support  of  U.S.  national 
defense,  homeland  security,  and  safety  of  navigation.  Presently 
headquartered  in  Bethesda,  Maryland,  with  principal  facilities  based 
in  the  St.  Louis,  Missouri,  and  Washington,  DC,  metro  areas,  NGA 
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is  in  the  process  of  consolidating  its  National  Capital  Region  facili¬ 
ties  to  comply  with  a  Base  Realignment  and  Closure  (BRAC)  2005 
decision. 

BRAC  2005  Recommendation  168,  which  was  enacted  into  law 
in  November  2005,  directed  the  following  activity: 

Close  National  Geospatial-Intelligence  Agency  (NGA)  Dale- 
carlia  and  Sumner  sites,  Bethesda,  MD;  Reston  1,  2,  and  3 
leased  installations  in  Reston,  VA;  Newington  buildings  8510, 
8520,  and  8530,  Newington,  VA;  and  Building  213,  a  leased 
installation  at  the  South  East  Federal  Center,  Washington, 
DC.  Relocate  all  functions  to  a  new  facility  at  Fort  Belvoir, 
VA.  Realign  the  National  Reconnaissance  Office  facility, 
Westfields,  VA,  by  relocating  all  NGA  functions  to  a  new 
facility  at  Fort  Belvoir,  VA.  Consolidate  all  NGA  National 
Geospatial-Intelligence  College  functions  on  Fort  Belvoir 
into  the  new  facility  at  Fort  Belvoir,  VA.  (DoD,  2005) 

NGA  responded  by  establishing  an  NGA  Campus  East  (NCE) 
Program  Management  Office  (PMO)  early  in  2006  and  immediately 
developed  a  plan  to  meet  this  BRAC  mandate  (NGA,  2010).  While 
these  initial  efforts  were  underway,  Fort  Belvoir  updated  its  facility 
Master  Plan  and  completed  an  Environmental  Impact  Statement  to 
address  how  NGA-  and  other  BRAC-impacted  organizations  would 
be  accommodated  at  Fort  Belvoir  (U.S.  Army  Corps  of  Engineers 
[USACE],  2007).  Both  called  for  locating  NGA  at  Fort  Belvoir’s 
Engineer  Proving  Ground  (a  site  adjacent  to  1-95  in  Springfield,  that 
has  since  been  renamed  the  Fort  Belvoir  North  Area  [FBNA]),  and 
with  the  signing  of  the  Record  of  Decision  on  Aug.  7,  2007,  FBNA 
was  officially  designated  as  the  future  home  for  NGA. 

Program  Scope 

The  NCE  effort  included  facility,  information  technology  (IT), 
security,  and  deployment  as  primary  executing  elements.  Focusing 
on  the  facility  component,  its  scope  called  for  the  design,  construc¬ 
tion,  and  commissioning  of  a  2.4  million  gross  square  foot  (gsf) 
campus  able  to  accommodate  8,500  personnel.  As  the  initial  design 
took  shape,  these  requirements  were  satisfied  with  a  Main  Office 
Building  (MOB),  Central  Utility  Plant  (CUP),  Technology  Center 
(TC),  Garage  (structured  parking),  Visitor  Control  Center  (VCC),  and 
Remote  Inspection  Facility  (RIF)  (NGA,  2009a).  The  MOB  (indicated 
as  structure  “1”  in  Figure  1)  consists  of  two  8-story  office  buildings, 
each  roughly  900  feet  long  with  1  million  gsf  of  capacity,  and  con¬ 
nected  by  an  enclosed  atrium  structure.  The  CUP  (structure  “4”) 
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FIGURE  1.  NGA  CONSOLIDATION 


is  approximately  89,000  gsf  and  houses  the  utility  services  that 
are  distributed  to  the  campus  facilities.  The  TC  (structure  “2”)  is  a 
4-story  structure  roughly  140,000  gsf  in  size.  The  Garage  (structure 
“3”)  is  a  6-level  pre-cast  concrete  structure  providing  5,100  parking 
spaces  (in  compliance  with  the  National  Capital  Planning  Commis¬ 
sion  guidelines).  The  VCC  (structure  “5”)  is  an  8,300  gsf  facility 
located  on  the  campus  perimeter  and  allowing  access  control  over 
visitors.  The  RIF  (not  depicted  in  Figure  1)  is  a  separate  10,000  gsf 
structure  located  adjacent  to  a  main  access  point  to  FBNA;  it  allows 
for  security  screenings  of  all  inbound  deliveries  to  the  NCE. 

Facility  Acquisition  Strategy 

The  NCE  effort  is  an  enormous  undertaking,  and  due  to  the 
language  of  the  BRAC  directive,  not  only  did  the  facility  need  to 
be  designed,  constructed,  and  commissioned  by  the  mandated 
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FIGURE  2.  FACILITY  ACQUISITION  STRATEGIES 
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deadline,  but  the  requisite  enterprise  IT  architecture  and  security 
management  systems  had  to  be  designed,  installed,  tested,  and 
placed  into  operation;  and  NGA  had  to  deploy  8,500  personnel  and 
the  missions  they  performed  at  the  NGA  legacy  sites  to  the  NCE  by 
the  September  15,  2011,  suspense.  Assessing  early  schedules  of  the 
program’s  activities  revealed  that  the  facility  effort  was  on  the  criti¬ 
cal  path  and  key  to  the  program’s  success.  As  the  program’s  other 
efforts  were  dependent  on  the  facility  being  in-place,  an  acquisi¬ 
tion  strategy  had  to  be  determined  that  would  deliver  the  facility 
component  as  rapidly  as  possible. 

When  acquiring  facilities  through  new  construction,  the  acqui¬ 
sition  strategy  (Figure  2)  typically  follows  one  of  two  forms:  a 
Design-Bid-Build  (D-B-B),  or  a  Design-Build  (D-B).  In  the  case  of  a 
D-B-B,  a  facility  designer  is  contracted,  a  design  is  completed,  and 
then  a  construction  contract  is  awarded  to  build  the  facility.  A  D-B-B 
strategy  employs  sequential  activities  and  usually  represents  the 
longest  amount  of  time  to  deliver  a  usable  facility.  A  D-B  strategy 
calls  for  a  single  contract  that  awards  both  a  design  and  construc¬ 
tion  scope.  The  project  time  (duration)  savings  occur  not  only  from 
a  single  contract  source  selection  (vice  two  in  a  D-B-B),  but  also 
potentially  from  the  selected  contractor’s  ability  to  integrate  its 
design  and  construction  efforts  (this  second  variant  is  sometimes 
referred  to  as  a  D-B  “Fast  Track”). 
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At  the  onset,  the  NCE  PMO  settled  upon  a  D-B  facility  acquisi¬ 
tion  strategy.  As  the  architect  progressed  toward  a  35  percent  level 
of  design  that  would  be  used  to  secure  a  D-B  contractor  to  com¬ 
plete  the  project,  the  NCE  PMO  recognized  that  if  it  were  to  maintain 
this  course,  the  facility  might  be  completed  in  time,  but  even  with 
incremental  acceptances  of  completed  work,  minimal  time  was 
allotted  to  complete  the  remaining  scope  of  the  program  prior  to 
the  BRAC  suspense.  The  NCE  PMO  and  the  Baltimore  District  of 
USACE  agreed  upon  an  alternate  strategy— that  of  Early  Contractor 
Involvement  (ECI).  Similar  to  the  Construction  Manager  at  Risk  (CM 
@  Risk)  strategy  gaining  usage  in  the  private  and  commercial  sec¬ 
tors,  ECI  calls  for  the  awarding  of  separate  design  and  construction 
contracts,  with  the  construction  contract  award  occurring  very  early 
in  the  design  development  (at  a  10  to  15  percent  development  of 
the  design).  This  strategy  maximizes  the  construction  contractor’s 
ability  to  influence  the  design  itself  and  the  packaging  of  design 
elements  to  facilitate  a  rapid  initiation  of  construction  efforts  (Peck, 
Stuban,  Bagshaw,  &  Calloway,  2010). 

As  for  the  construction  contract  type,  given  the  relative  immatu¬ 
rity  of  the  design  and  a  need  to  control  cost,  a  “Fixed  Price  Incentive 
with  Successive  Targets”  format  was  chosen  in  accordance  with 
Federal  Acquisition  Regulation  52.216-17.  Doing  so  allowed  for 
establishment  of  target  and  ceiling  prices  for  the  various  elements 
of  work  and  incentivizing  cost  containment  (Peck  et  al.,  2010). 

Program  Governance 

The  NCE  PMO  managed  the  totality  of  the  program  effort,  but 
executing  prime  contractors  were  controlled  by  an  assortment  of 
contract  management  teams,  many  of  which  were  external  to  NGA. 
The  facility  efforts  were  managed  by  the  USACE  Baltimore  District; 
the  security  management  system  and  construction  surveillance 
technician  contracts  were  managed  by  the  U.S.  Navy  Space  and 
Naval  Warfare  Systems  Command;  and  the  site  security,  IT,  and 
deployment  contracts  were  managed  by  NGA. 

In  addition  to  these  efforts  internal  to  the  program,  the  program 
was  also  dependent  on  the  substantial  efforts  of  a  number  of  ele¬ 
ments  external  to  NGA  and  the  NCE  PMO:  the  Virginia  Department 
of  Transportation  for  a  number  of  roadway  improvements  adja¬ 
cent  to  the  FBNA;  the  Fort  Belvoir  Garrison  staff  for  infrastructure 
improvements  to  the  FBNA;  commercial  utility  providers  for  gas  and 
electric  service  improvements  to  the  FBNA;  and  several  telecom¬ 
munications  providers  for  wide  area  network  connectivity. 

To  enhance  communication  and  coordination  between  these 
various  parties,  the  NCE  PMO  established  a  3-tiered  management 
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structure  termed  the  “One  Team”  (NGA,  2009b).  At  the  founda¬ 
tional  level,  the  Project  Leadership  Teams  (PLTs)  are  focused  on 
efforts  underway  at  their  discrete  project  level  (the  MOB,  CUP,  TO, 
etc.).  The  PLT  membership  consists  of  representatives  from  all  ele¬ 
ments  engaged  in  delivering  a  completed,  occupied,  and  operational 
project,  and  includes  facilities  (design  and  construction),  IT,  security, 
deployment,  Ft.  Belvoir  Garrison,  and  operations  and  maintenance 
staff.  Mid-level  governance  is  provided  by  an  Executive  Leadership 
Team  (ELT),  co-chaired  by  the  PMO’s  Deputy  Program  Director— Site 
and  Baltimore  District’s  program  manager  for  the  NCE  effort.  ELT 
membership  is  comprised  of  the  PMO’s  deputy  and  assistant  pro¬ 
gram  managers,  and  the  program/project  managers  (both 
government  and  contractor)  of  each  executing  element.  Top-level 
governance  is  provided  by  a  Program  Board  (PB),  co-chaired  by 
the  NCE  PMO’s  program  director  and  the  Baltimore  District  com¬ 
mander.  Like  the  ELT,  membership  consists  of  executives  (both 
government  and  contractor)  of  each  executing  element  (Figure  3). 

FIGURE  3.  NCE  GOVERNANCE  STRUCTURE 
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PLTs  meet  on  a  weekly  basis  (or  more  frequently  depending  on 
emergent  issues),  the  ELT  meets  biweekly,  and  the  PB  meets  once  a 
month.  Each  PLT  has  its  own  decision  space  and  authority.  So  long 
as  the  PLT’s  decisions  do  not  adversely  impact  another  program 
element,  perturb  a  program-level  milestone,  or  exceed  their  budget 
authority,  they  can  directly  manage  their  project’s  effort.  Activities 
that  may  adversely  impact  other  program  elements,  or  are  outside 
the  PLT’s  decision  space,  are  elevated  to  the  ELT  (or  PB  if  necessary) 
for  resolution  (NGA,  2009b). 

Risk  Management 

The  PMO  has  from  the  onset  of  the  program  employed  standard 
program  management  and  systems  engineering  technical  manage¬ 
ment  processes  to  execute  the  program  within  established  cost, 
schedule,  and  performance  constraints.  Many  of  the  techniques 
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FIGURE  4.  NCE  RISK  QUAD  TEMPLATE 
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employed  (requirements  management,  schedule  management, 
change  management,  etc.)  were  commonly  understood  by  all  mem¬ 
bers  of  the  One  Team  and  were  summarily  described  in  the  NCE 
Program  Management  Plan  (PMP).  When  the  NCE  program  was  initi¬ 
ated  in  2006,  risk  management  as  a  means  to  contain  cost,  maintain 
schedule,  or  ensure  performance  had  only  recently,  however,  been 
adapted  on  MILCON  projects.  Drawing  upon  NGA’s  enterprise  risk 
management  process  and  the  DoD’s  Risk  Management  Guide  for 
DoD  Acquisition,  the  PMO  crafted  a  Risk  and  Opportunity  Manage¬ 
ment  Plan  (ROMP)  and  tool  set,  approved  by  the  NCE  program 
director  (PD),  which  was  incorporated  into  the  NCE  PMP  and  used 
across  the  One  Team  to  facilitate  the  management  of  risk. 

A  Risk  and  Opportunity-focused  Integrated  Process  Team  (IPT) 
was  established.  Like  the  PLTs,  its  membership  included  representa¬ 
tion  from  all  of  the  program’s  executing  elements  and  is  facilitated 
by  the  PMO’s  government  and  contractor  Program  Integration 
staff.  Employing  standard  Microsoft  Office  applications,  the  IPT 
formalized  a  “Risk  Quad  Chart”  template  (Figure  4)  to  capture  the 
essential  elements  of  information  necessary  to  assess  a  potential 
risk,  opportunity,  or  issue. 
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Starting  with  the  upper  left  quadrant  and  moving  counter¬ 
clockwise,  an  “If-Then”  formatted  statement  is  utilized  to  identify 
the  root  cause  of  a  potential  future  situation,  which,  if  mitigated, 
would  preclude  a  potential  adverse  consequence.  The  “Decision/ 
Trigger  Points”  line  notes  when  decisions  may  need  to  be  made 
between  alternative  courses  of  action  or  in  furtherance  of  the 
intended  mitigation  plan.  The  “Closure  Criteria”  define  what  con¬ 
stitutes  successful  mitigation  of  the  potential  risk.  The  “Mitigation 
Plan”  section  chronologically  outlines  the  discrete  steps  to  be 
taken  in  mitigating  the  potential  risk.  The  “Context”  line  provides 
for  further  background  information  as  to  the  development  of  the 
potential  risk.  The  “Status”  line  allows  for  entry  of  relevant  recent 
information.  The  “Risk  Score  with  Analysis”  allows  for  entering  the 
assessed  “probability”  or  likelihood  of  the  risk  occurring  as  well 
as  the  consequence  or  adverse  impact  assessments  from  a  cost, 
schedule,  and  performance  perspective. 

The  Risk  Scoring  is  based  on  a  standard  5x5  matrix  and  prob¬ 
ability  definitions/percentages  (DoD,  2006).  The  consequence 
definitions  are  specific  to  the  NCE  program. 

Similar  quad  charts  and  scoring  rubrics  were  developed  for 
assessing  “opportunities”  (potential  future  conditions  that,  if 
exploited,  could  result  in  positive  consequences  for  the  program) 
and  “issues”  (existing  conditions  that  were  having  an  adverse  impact 
on  the  program). 

With  development  of  the  ROMP  and  tool  sets,  and  conduct  of 
refresher  training,  the  PLTs  were  allowed  to  manage  risks,  issues, 
and  opportunities  at  their  level.  If  the  PLTs  determine  that  addi¬ 
tional  resources  may  be  required  to  successfully  mitigate  a  risk  or 
if  mitigation  is  outside  their  defined  decision  space,  the  risk  has  to 
be  coordinated  via  the  Risk  IPT  and  elevated  within  the  program. 

The  Risk  IPT  meets  on  a  biweekly  basis  and  serves  as  the  forum 
in  which  anyone  associated  with  the  program  could  suggest  an 
NCE  program-related  risk,  issue,  or  opportunity.  The  IPT  consid¬ 
ers  suggested  matters  and  aids  in  drafting  an  associated  quad 
chart.  Once  drafted  and  coordinated  across  the  IPT’s  membership, 
the  IPT  determines  what  recommendation  should  be  made  to  the 
program’s  Risk  and  Opportunity  Management  Board  (ROMB).  The 
ROMB  meets  monthly  and  is  chaired  by  the  NCE  PMO  PD.  The  PD 
is  briefed  on  the  proposed  risk  and  the  IPT’s  recommendation,  and 
then  renders  a  decision  as  to  whether  the  risk  should  be  placed  in 
a  “watch”  status  (to  allow  for  validation  of  the  potential  conditions 
that  are  suspected),  “opened”  and  actively  mitigated,  elevated 
to  NGA’s  enterprise-level  risk  management  board,  or  returned  to 
the  IPT  for  further  coordination.  Risks  that  are  opened,  elevated, 
or  placed  on  a  watch  status  are  then  tracked  in  a  Risk  Register  (a 
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FIGURE  5.  COST  GROWTH 


NCE 
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Project 
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(Base  +  Options) 

Cost 

Growth 

Amount 

Final 

Contract 

Amount 

Cost 

Growth 

(%) 

CUP 

$99,961,243 

$7,503,144 

$107,464,387 

7.51 

TC 

$77,996,108 

($5,645,069) 

$72,351,039 

(7.24) 

Garage 

$76,729,943 

($4,443,661) 

$72,286,282 

(5.79) 

VCC 

$5,880,734 

$79,397 

$5,960,131 

1.35 

spreadsheet  chronologically  detailing  every  mitigation  step  of  every 
risk)  and  statused  monthly  at  the  ELT  and  PB  sessions. 

To  date  the  NCE  PMO  has  handled  nearly  150  separate  risks, 
issues,  and  opportunities  above  the  PLT  level. 

Cost  Growth  Record 

Have  the  NCE  program’s  active  risk  management  activities 
made  any  difference  in  the  cost  growth  realized  on  the  MILCON 
component  of  the  program?  To  assess  this  possibility,  a  t-test  for 
independent  samples  was  performed  (Salkind,  2009).  In  this  test, 
a  comparison  was  made  between  the  cost  growth  realized  on 
several  of  the  NCE  program’s  facility  projects  that  were  at  or  near 
a  substantial  completion  point  and  the  cost  growth  realized  on  a 
sample  of  USACE  MILCON  projects  completed  prior  to  FY06  (a 
timeframe  when  active  risk  management  as  employed  on  the  NCE 
program  was  not  practiced)  (J.  J.  Tyler,  personal  communication, 
July  13,  2009). 

MILCON  projects  completed  by  USACE  in  FY04  and  FY05  were 
assessed  (earliest  complete  fiscal  year  data  available  from  USACE) 
(USACE,  2010).  From  this  sample  set,  projects  completed  outside 
the  continental  United  States  (CONUS)  were  excluded  due  to  exter¬ 
nal  impacts  that  could  influence  the  true  cost  growth  (material 
shipping  costs,  material  and  labor  availability,  currency  exchange 
rate  fluctuations,  construction  in  military  theaters  of  operation, 
etc.).  This  yielded  15  projects  completed  in  FY04  and  38  projects 
completed  in  FY05  (a  total  sample  size  of  53),  ranging  in  value  from 
roughly  $1.4  million  to  nearly  $45  million.  Comparing  each  project’s 
baseline  contract  and  options  amount  to  its  final  contract  amount 
(determined  after  all  construction  was  complete  and  the  contract 
was  financially  closed-out)  revealed  the  cost  growth  realized  on  the 
projects.  Assessing  the  cost  growth  on  all  53  projects  revealed  a 
sample  mean  cost  growth  equaling  7.493  percent,  with  a  standard 
deviation  of  9.728. 
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Four  NCE  facility  projects  were  included  in  the  comparison 
sample:  the  CUP,  TC,  Garage,  and  VCC.  The  financial  details  for 
these  projects  are  shown  in  Figure  5. 

Assessing  the  cost  growth  of  the  four  projects  listed  in  Figure 
5  revealed  a  sample  mean  cost  growth  equaling  -1.04  percent  with 
a  standard  deviation  of  6.824. 

Translating  this  into  cost  totals,  had  the  NCE  program  experi¬ 
enced  the  average  cost  growth  of  the  historical  sample,  it  would 
have  incurred  $22,234,270  in  additional  costs. 

The  NCE  program  unquestionably  managed  to  better  control 
costs  relative  to  that  of  the  historical  sample.  To  assess  whether  it 
was  statistically  significant— that  the  NCE  sample  was  indeed  dif¬ 
ferent  from  the  historical  sample  and  not  simply  an  outlier— a  t-test 
for  independent  samples  was  performed.  The  NCE  sample’s  t0  was 
calculated  as  1.715  (Walpole  &  Myers,  1978).  This  t0  value  was  plot¬ 
ted  on  a  t-distribution  of  the  historical  sample.  The  distribution’s 
*00555  value  equals  1.673;  this  is  the  point  at  which  95  percent  of 
the  distribution  with  the  appropriate  55  degrees  of  freedom  lies 
to  the  left.  Focusing  on  this  point  revealed  that  t0  in  this  case  lies 
to  the  right  (it  is  in  the  critical  zone).  This  signifies  that  the  mean 
cost  growth  realized  on  the  NCE  projects  is  statistically  significant 
relative  to  that  of  the  USACE  sample  of  FY04  and  FY05  (Salkind, 
2009).  As  the  NCE  cost  growth  is  lower  than  that  of  the  USACE 
sample,  it  is  preferred,  and  whatever  characteristic(s)  made  the 
NCE  sample  distinct  from  the  USACE  sample  would  be  preferred 
as  well.  It  is  suggested  that  an  active  risk  management  process  is 
at  least  one  of  the  characteristics  that  sets  the  NCE  projects  apart 
from  the  way  historical  MILCON  projects  have  been  managed,  and 
is  a  process  that  should  be  employed  on  all  MILCON  projects  (if  not 
already  underway)  where  controlling  cost  growth  is  an  objective. 

Conclusions 

Risk,  issues,  and  opportunities  are  ever-present  and  require 
proactive  management  approaches  throughout  an  acquisition  to 
ensure  that  a  program’s  cost,  schedule,  and  performance  objectives 
are  met.  In  that  DoD  acquisition  takes  many  forms,  including  facility 
acquisition  via  MILCON  projects,  leveraging  all  the  management 
tools  and  techniques  that  may  be  available  appears  to  be  the  most 
prudent  course  of  action.  An  active  risk  management  program,  par¬ 
ticularly  applied  throughout  the  project’s  delivery  phase  (the  design, 
construction,  and  commissioning  of  the  facility),  is  one  such  tool. 
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Government  contracting  is  rife  with  opportunities  for  miscom- 
munication  and  misperception.  This  can  undermine  trust  and 
fuel  spirals  of  conflict.  For  this  article,  the  authors  interviewed 
participants  and  analyzed  Government  Accountability  Office 
(GAO)  bid  protest  decisions  involving  Department  of  Defense 
source  selections.  They  found  agency,  vendor,  and  GAO  prac¬ 
tices  that  trigger  and  fuel  these  spirals.  Contracting  agencies 
and  GAO  can  take  steps  to  improve  communication,  reduce 
inconsistencies,  and  reduce  perceptions  of  bias,  thereby  miti¬ 
gating  costly  bid  protests. 
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In  1984,  Congress  gave  its  investigatory  arm,  the  General 
Accounting  Office,  or  GAO  (renamed  the  Government  Account¬ 
ability  Office  in  July  2004),  authority  to  decide  protests  of  source 
selection  decisions  under  the  Competition  in  Contracting  Act. 
Judicial  forums  for  resolving  protests,  such  as  the  Court  of  Federal 
Claims  (COFC)  or  Circuit  Courts,  are  adversarial  by  design.  GAO 
is  not.  It  is  an  alternative  dispute  resolution  (ADR)  mechanism, 
designed  to  be  fast,  inexpensive,  and  flexible. 

Flow  well  does  this  process  work?  The  response  to  this  ques¬ 
tion  evokes  not  one,  but  two  answers.  If  the  question  means: 
Does  it  handle  protests  efficiently?  the  answer  is  clearly  “yes.”  If 
it  means:  Does  it  fix  the  root  causes  of  the  conflicts  that  lead  to 
protests?  the  answer  is  less  clear.  To  answer  the  question,  we  apply 
concepts  from  the  theory  of  dispute-systems  design  (Costantino 
&  Merchant  1996;  Slaikeu  &  Hasson,  1998;  Stitt,  1998;  Lynch,  2001; 
Conbere,  2001;  Lipsky,  Seeber,  &  Fincher,  2003;  Shariff,  2003; 
Bordone,  2008). 

During  Fall  2009  and  Winter  2010,  we  used  a  protocol  designed 
to  diagnose  conflict  (Ury,  Brett,  &  Goldberg,  1988)  to  interview  over 
25  members  of  the  acquisition  community: 

•  Four  attorneys  at  GAO 

•  Executives  and  in-house  counsel  at  four  prime 
contractors 

•  Four  outside  bid  protest  counsel 

•  Contract  managers  at  two  small  subcontractors 

•  Current  and  former  officials  in  the  Office  of  the  Secre¬ 
tary  of  Defense 

•  Officials  and  in-house  attorneys  at  three  military  com¬ 
mands— Air  Force  Materiel  Command,  Naval  Air  Systems 
Command,  and  Defense  Logistics  Agency 

•  Senate  Committee  staff 

•  Executives— typically,  former  DoD  contracting  officers 
with  industry  trade  or  professional  associations  such 
as  the  Aerospace  Industries  Association,  the  National 
Contract  Management  Association,  the  Professional 
Services  Council,  and  TechAmerica. 

These  are  not  a  representative  sample,  but  rather  a  network  that 
expanded  as  respondents  recommended  others  who  could  share 
their  perspectives.  Their  insights  are  suggestive,  not  definitive. 

We  also  analyzed  GAO  bid  protest  decisions  related  to  DoD 
source  selections  between  2001  and  2009.  Our  interlocutors  gener¬ 
ally  agree  that  source  selection  procedures,  although  often  onerous 
for  everyone  involved,  are  basically  fair  and  bid-protest  processes 
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effective.  They  also  expressed  concern  about  insufficient  informa¬ 
tion,  inconsistency,  and  bias. 

Insufficient  Information 

In  the  absence  of  information,  innocuous  matters  can  grow  into 
spirals  of  conflict.  Miscommunication  and  misperception  trigger 
distrust  and  sometimes  hostility  even  though  the  procedures  for 
resolving  disagreements  seem  clear  (Carpenter  &  Kennedy,  2001). 
For  example,  an  offeror,  having  made  a  significant  investment  in 
the  process,  seeks  information  or  acknowledgement  of  a  problem. 
The  contracting  agency,  also  having  made  a  significant  investment, 
resists.  Negotiating  does  not  resolve  the  problem.  The  offeror  per¬ 
ceives  the  agency  to  be  stonewalling.  The  agency  perceives  the 
offeror  to  be  seeking  a  competitive  advantage. 

After  protesting  informally,  an  offeror  might  protest  formally 
within  the  contracting  agency  or  skip  the  formal,  agency-level 
review  and  go  directly  to  GAO.  Other  parties  begin  to  take  sides. 
Elected  officials,  for  example,  step  in,  perhaps  directing  affected 
constituents  to  pursue  the  protest  at  the  GAO.  The  contract  winner 
may  step  in  to  support  the  agency. 

GAO  procedures  are  fairly  well  defined  and  managed,  often 
resolving  the  dispute.  However,  a  company  dissatisfied  with  GAO’s 
decision  can  go  to  COFC  or  pursue  the  matter  in  Congress  or 
with  other  decision  makers  at  DoD  or  elsewhere  in  the  Executive 
Branch— a  relatively  unmanaged  process.  As  the  conflict  escalates, 
communication  becomes  fraught;  misunderstandings  multiply.  Zeal¬ 
ots  replace  moderates  and  invest  resources  to  win  rather  than  to 
resolve  disagreements.  Perceptions  distort,  parties  lose  objectivity, 
gray  areas  become  black  or  white,  or  seemingly  innocuous  behav¬ 
iors  become  meaningful  as  distrust  and  suspicion  grow. 

To  generate  more  complete  and  accurate  information,  govern¬ 
ment  establishes  regulations,  typically  in  the  Federal  Acquisition 
Regulation  (FAR)  and  Defense  Federal  Acquisition  Regulation 
Supplement  (DFARS).  In  the  Federal  Acquisition  Streamlining  Act  of 
1994,  Congress  made  agency  debriefings  mandatory  upon  request. 
Rejected  offerors  may  request  information  from  the  agency  about 
the  basis  for  its  selection  decision  and  contract  award.  A  potential 
benefit  of  conducting  a  debriefing  is  to  prevent  a  bid  protest  by 
explaining  the  reason  for  agency  decisions  so  the  rejected  offeror 
will  see  that  the  agency  acted  within  the  bounds  of  its  discretion 
and  consistent  with  its  evaluation  plan. 

If  an  agency  gears  its  standard  of  disclosure  to  surviving  the 
protest  at  GAO,  which  can  result  in  the  agency  sharing  less  infor¬ 
mation,  the  offeror,  anticipating  this,  might  start  bringing  attorneys 
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to  debriefings  to  elicit  more  information.  The  engineers,  attorneys, 
or  head  of  a  business  unit  want  to  explain  to  the  team  that  spent 
time  working  on  a  proposal  why  the  company  lost.  Executives  to 
whom  they  report  want  to  know,  as  well.  The  agency  might  perceive 
the  presence  of  attorneys  as  a  threat  (Szeliga,  2008).  In  a  classic 
illustration  of  a  conflict  spiral,  the  dissatisfied  offeror  files  a  protest 
and  contracting  agency  executives  have  to  explain  to  their  team, 
who  also  invested  time  working  on  the  source  selection,  why  the 
company  filed  a  protest,  and,  potentially,  why  GAO  sustained  it. 

Where  an  agency  discloses  in  a  debriefing  as  much  to  rejected 
offerors  as  it  would  to  the  Source  Selection  Authority,  some  offerors 
will  be  grateful  and  satisfied.  However,  some  rejected  offerors  will 
comb  the  information  to  find  bases  for  challenges.  A  business  con¬ 
sultant  and  contactor  said,  “Even  if  you  give  a  contracting  officer  a 
script  for  the  debriefing,  written  by  an  attorney,  a  rejected  offeror 
can  find  a  problem  in  a  gesture  or  a  phrase.”  A  prime  contractor 
executive  said,  “If  the  agencies  are  becoming  paranoid  because 
attorneys  are  involved  earlier  so  agency  people  become  more 
cautious  in  what  they  say,  remember  the  old  saying:  ‘Just  because 
you’re  paranoid  doesn’t  mean  someone  isn’t  out  to  get  you.’” 

Agencies  fear  rejected  offerors  will  exploit  their  every  word,  so 
utter  fewer  of  them.  Businesses  fear  agencies  will  utter  fewer  words, 
so  try  to  pry  more  out  of  them.  In  a  spiral  of  conflict,  perception 
matters  more  than  substance.  Reciprocating  reactions  create  an 
adversarial  tone. 


Inconsistencies 

Ironically,  postaward  debriefings  can  contribute  to  a  climate  of 
distrust  because  the  FAR  gives  a  contracting  officer  discretion  in 
the  content  of  debriefings.  At  one  agency,  a  vendor  might  receive  a 
10-minute  review,  scripted  by  an  agency  attorney,  with  a  contract¬ 
ing  official  showing  one  PowerPoint  slide  containing  the  minimal 
amount  of  information  required  by  the  FAR  and  minimal  opportunity 
for  the  rejected  offeror  to  ask  questions.  At  a  second,  the  vendor 
might  receive  an  analysis  of  what  the  contractor  did  or  did  not  do 
that  was  problematic.  At  a  third,  the  vendor  might  receive  a  2-day 
review  by  multiple  members  of  the  source  selection  team,  including 
engineers  and  attorneys,  presenting  essentially  the  same  informa¬ 
tion  conveyed  to  the  Source  Selection  Authority;  the  agency  will 
ask  the  winner  for  permission  to  explain  to  rejected  offerors  why 
the  agency  selected  the  winner,  albeit  with  competitive  informa¬ 
tion  redacted.  The  rejected  offeror  has  ample  opportunity  to  ask 
questions.  Even  within  the  same  agency,  people  disagree  on  which 
debriefing  approach  to  implement. 
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Not  knowing  what  they  will  encounter,  businesses  prepare  for 
the  worst.  Conversely,  while  some  business  executives  maintain 
resolutely  that  they  attend  debriefings  to  find  reasons  not  to  protest, 
agencies  cannot  necessarily  discriminate  them  from  executives  who 
attend  to  prepare  to  protest.  Reflecting  the  cost  of  inconsistency,  a 
protest  attorney  believes  agencies  build  3  months  into  their  sched¬ 
ules  for  large  contracts  to  account  for  bid  protests,  and  companies 
build  the  expected  cost  of  a  protest  into  their  overhead. 

After  companies  file  protests,  disclosure  practices  also  can  be 
inconsistent  across  agencies.  Anticipating  a  protest,  one  agency 
might  have  documented  every  step  it  took  from  the  outset  and 
prepared  to  reveal  all.  Another  might  not  create  a  file,  as  in  a  legal 
discovery  process,  until  the  protest  has  been  filed.  If  the  bid  protest 
targets  a  particular  part  of  the  selection  process,  an  agency  might 
focus  its  disclosure  on  only  the  protested  part.  If  a  rejected  offeror 
is  unable  to  distinguish  an  agency  that  will  disclose  more  postpro¬ 
test  from  one  that  will  disclose  less,  it  has  an  incentive  to  challenge 
multiple,  interrelated  parts  and,  prior  to  that,  to  mine  debriefings 
for  information  that  could  provide  the  basis  for  protests,  increasing 
the  costs  to  the  agency  and  irritating  its  decision  makers. 

A  perception  of  inconsistency  afflicts  decision  making  at  GAO, 
as  well.  GAO  attorneys  discriminate  frivolous  from  legitimate 
protests— those  that  point  out  an  error  in  a  contracting  agency’s 
processes.  They  also  differentiate  among  legitimate  claims  those 
that  are  material— meaning  the  outcome  of  the  source  selection 
might  have  been  different  but  for  the  agency’s  error— from  those 
that  are  immaterial.  In  that  sense,  GAO,  in  effect,  applies  a  standard 
of  reasonableness  in  its  bid  protest  decisions  and  works  to  maintain 
that  standard  with  consistency. 

Members  of  the  acquisition  community  on  both  the  government 
and  the  business  side  believe  GAO’s  standards  of  reasonableness 
and  materiality  have  eroded,  encouraging  more  protestors  to  file 
protests  and  more  protests  to  involve  frivolous  and  immaterial 
claims.  GAO  disagrees  that  its  standard  of  reasonableness  has 
declined.  An  independent  legal  analysis  might  confirm  that  it  has 
not.  What  nourishes  spirals  of  conflict,  however,  is  the  perception 
that  it  has. 

Similarly,  some  agencies  and  legal  practitioners  expect  GAO 
to  follow  precedent  but  perceive  that  it  does  not.  Others  believe 
GAO  exercises  discretion  in  the  areas  where  it  chooses  to  rule  and 
on  the  direction  of  its  rulings,  by  ignoring  facts  in  one  case  that 
are  the  same  as  in  another  case  and  should  be  determinative  in 
both.  When  a  new  area  of  dispute  arises,  such  as  organizational 
conflicts  of  interest  in  the  early  2000s,  GAO  will  choose  to  find 
merit  in  claims  in  the  new  area  and  begin  sustaining  them  until  the 
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acquisition  community  adapts  to  the  likelihood  of  those  claims  pre¬ 
vailing.  A  protest  attorney  might  argue  less  on  precedent  relevant 
to  the  main  issue  in  a  protest  than  on  attracting  GAO’s  attention 
to  a  minor  issue  that  could  set  a  new  one.  For  good  or  ill,  GAO’s 
pursuit  of  its  multiple  missions— third-party  intervener,  educator, 
and  promoter  of  competition— can  contribute  to  a  perception  of 
its  inconsistency. 


Perceptions  of  Bias 

The  acquisition  community’s  perception  of  bias  takes  several 
forms.  For  example,  few  of  our  business  interviewees  found  agency- 
level  reviews  to  be  efficacious  (Troff,  2005)  because  they  believe 
agencies  become  defensive.  A  bid-protest  attorney  asks  why  an 
agency  would  correct  its  own  mistake?  “It’s  more  likely  to  circle  the 
wagons.  An  agency  review  is  a  single  filing,  no  discovery,  and  you 
wait  for  an  agency  to  decide.”  He  recommends  against  it. 

An  informal  agency  review  involving  one-on-one  conversations 
can  make  an  agency  nervous  that  a  company  will  try  to  influence 
the  definition  of  requirements  or  the  evaluation  scheme  to  favor 
the  proposal  it  intends  to  submit.  One  trade  association  official 
described  this  as  “a  Kabuki  dance.”  After  agencies  publish  solicita¬ 
tions,  businesses  believe  the  agencies  have  vested  interests  in  them 
and  tend  to  be  dismissive  of  inquiries  from  companies.  Frustrated 
offerors  do  not  see  agencies  as  neutral  venues. 

Vendors  hear  agencies  say  that  protests  are  part  of  the  source 
selection  process  and  they  do  not  and  legally  may  not  treat  a 
protestor  with  prejudice  during  a  subsequent  selection.  But,  as  a 
consultant  to  many  offerors  put  it:  “The  contracting  community 
lives  in  fear  of  retribution  for  protesting.”  A  vendor  protests,  then 
loses  a  subsequent  contract  and  attributes  the  failure  not  to  its 
unresponsive  bid,  but  to  the  agency  seeking  retribution.  Or,  vendors 
experience  retribution  for  poor  performance  in  the  business  world 
and  project  it  into  the  government  world.  Offerors  believe  that 
protests  impact  careers  in  the  agency,  leaving  its  decision  makers 
with  prejudice. 

The  business  executives’  fears  are  not  necessarily  misplaced.  A 
former  contracting  official  described  the  ease  with  which  an  agency 
can  exact  revenge.  Suppose  a  contracting  official  wishes  to  punish  a 
vendor  who  protested  and  subsequently  plans  to  bid  on  a  contract 
to  be  performed  outside  its  geographical  area  in  competition  with 
vendors  close  to  the  location  of  performance.  The  contracting  offi¬ 
cial  specifies  in  the  solicitation  that  expenses  will  not  be  reimbursed 
for  travel  in  excess  of  50  miles,  effectively  denying  the  target  offeror 
an  opportunity  to  bid. 
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Some  members  of  the  acquisition  community  perceive  that 
Democratic  administrations  favor  particular  firms,  Republicans 
others,  that  defense  agencies  have  their  pets,  and  that  the  GAO 
decisions  reflect  congressional  preferences.  A  few  cited  specific 
examples  that  confirmed  their  suspicions,  but  most  were  based  on 
little  more  than  hearsay.  What  is  remarkable  about  these  responses 
is  the  distrust  the  participants  expressed  about  the  source  selection 
process,  despite  the  fact  that,  when  queried  about  their  own  expe¬ 
riences,  they  often  described  the  officials  they  had  direct  contact 
with  as  open,  helpful,  and  informative. 

Several  factors  bear  on  this.  First,  a  rejected  offeror,  not  hav¬ 
ing  achieved  its  objectives,  will  blame  the  process.  This  is  human 
nature,  a  “self-serving  attribution”  (Malhotra  &  Bazerman,  2007, 
p.  135).  As  a  trade  association  official  put  it:  “When  you’ve  lost, 
you  distrust  the  system  and  believe  the  decision  was  wired  for 
someone  else.” 

Second,  the  inherent  subjectivity  of  the  decisions  made  by 
agencies  induces  distrust  of  the  process  by  business  participants. 
Evaluating  “best  value,”  for  example,  requires  balancing  price, 
performance,  and  other  characteristics,  which  is  problematic.  A 
contracting  agency  official  said:  “Even  big  companies  believe  gov¬ 
ernment  looks  only  at  lowest  price,  not  at  best  value.”  Business 
executives  concur  (Schofield,  2009,  p.  53). 

Third,  smaller  companies,  who  comprise  the  majority  of  offerors 
and  a  disproportionate  source  of  protests,  are  less  sophisticated. 
They  might  not  devote  resources  to  obtaining  contracting  expertise 
or  in-house  or  outside  counsel.  The  company  errs  but  believes  the 
government  did.  A  small  company  might  protest  because  it  believes 
an  injustice  has  been  done.  Indeed,  it  might  perceive  a  bias  based 
solely  on  its  size,  a  view  expressed  by  a  business  executive  at  a 
smaller  firm  who  said,  “No  one  gets  fired  for  hiring  Raytheon,  but 
someone  can  get  fired  for  hiring  [my  company].”  In  contrast,  a  large 
company  with  multiple  product  lines  makes  a  business  decision  to 
protest  based  on  assessing  the  potential  outcome  versus  the  cost 
of  pursuing  the  protest. 

Given  the  high  cost  of  understanding  these  processes,  a  visible, 
sustained  protest  on  a  high-value  contract,  like  the  KC -Tanker,  sends 
a  signal  throughout  the  contracting  community,  triggering  new  spi¬ 
rals.  According  to  a  bid  protest  attorney:  “Lots  of  contractors  now 
think  that  if  they  work  hard,  turn  in  a  good  bid,  and  protest  vigor¬ 
ously,  they  might  win,  as  Boeing  did.  [My  firm]  has  handled  twice 
as  many  protests  during  the  past  two  years  as  in  the  previous  two.” 
Decision  makers  assign  outsized  significance  to  low-probability 
events  with  significant  impact,  like  a  sustained  protest  on  a  high- 
value  contract. 
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Fourth,  companies  create  advantages  for  themselves,  some¬ 
times  in  ways  that  undermine  confidence  in  the  contracting  process. 
For  example,  a  company  buys  expertise  about  the  contracting  pro¬ 
cess  by  recruiting  contracting  officers  from  government  agencies. 
Competitors  believe  that  these  contracting  officers  will  trade  not 
only  on  their  expertise,  but  also  on  their  relationships  with  decision 
makers  in  the  contracting  command. 

Fifth,  contracting  commands  need  expertise  from  their  suppliers 
to  define  requirements.  Not  all  suppliers  have  the  access,  experi¬ 
ence,  and  resources  to  respond.  The  result  can  be  requirements  that 
preclude  some  suppliers  from  qualifying.  Regulations  designed  to 
create  fairness  can  have  the  opposite  effect  because  of  their  com¬ 
plexity.  According  to  a  business  executive:  People  who  know  how 
to  play  the  game  will  prevail. 

Sixth,  although  GAO  maintains  that  it  operates  on  professional 
principles,  immune  from  political  influence  by  members  of  Congress, 
people  in  the  business  world  do  not  believe  it.  A  legal  practitioner 
at  a  prime  contractor  said,  “No  matter  what  the  issue,  you  can  find 
GAO  opinions  on  either  side.  GAO  tries  to  keep  the  politics  out  of 
it.  I  don’t  know  how  they  do  it  when  their  bosses  in  Congress  are 
calling  them  in  to  testify  at  hearings.” 

Businesses  seek  congressional  assistance  in  securing  a  contract 
or  in  protesting  failure  to  win.  GAO  believes  members  of  Congress 
like  being  able  to  direct  their  constituents  to  GAO  for  a  neutral  hear¬ 
ing,  rather  than  having  to  do  battle  over  the  matter  with  another 
member  (McCubbins  &  Schwartz,  1984).  Nonetheless,  as  elected 
officials  are  wont  to  do,  they  will  take  credit  for  GAO  decisions  that 
favor  their  constituents,  damaging  perceptions  of  GAO’s  neutrality. 

Recommendations 

In  general,  agency  officials  agree  with  industry  executives  and 
attorneys:  What  an  agency  does  to  conduct  a  good  source  selection 
is  also  what  will  avoid  a  protest.  However,  given  the  root  causes  we 
found  to  be  associated  with  spirals  of  conflict  in  source  selections, 
we  recommend  changes  that  increase  the  flow  of  information, 
improve  consistency,  and  reduce  perceptions  of  bias.  The  recom¬ 
mendations  implement  principles  of  dispute-systems  design  (Ury 
et  al .,  1988). 

Principle  No.  1:  Put  the  Focus  on  Interests 

To  short  circuit  a  spiral  of  conflict,  focus  the  parties  on  solving 
a  mutual  problem,  face-to-face,  in  a  relatively  informal  process  that 
they  help  to  shape  (Carpenter  &  Kennedy,  2001,  pp.  26-29).  The 
parties  at  the  lowest  level  will  have  the  best  information,  be  able  to 
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respond  most  quickly,  and  be  more  likely  to  satisfy  their  underlying 
interests— the  reasons  why  each  party  is  participating  in  the  source 
selection  process— so  they  are  less  likely  to  perceive  bias.  That 
would  be  agency-level  review. 

At  the  Defense  Logistics  Agency  (DLA),  contracting  officers 
respond  when  there  are  source  selection  problems.  Sometimes,  they 
educate  offerors.  Other  times,  if  an  offeror  is  correct,  the  contract¬ 
ing  officer  rectifies  the  situation.  If  an  offeror  formally  protests  to 
DLA,  it  can  choose  to  go  to  either  the  contracting  officer  or  to  the 


chief  of  the  contracting  officer,  but  not  both.  DLA’s  internal  alterna¬ 
tive  dispute  resolution  (ADR)  process  employs  trained  mediators. 
No  appeal  within  DLA  is  possible;  the  next  step  is  GAO.  Since  2004, 
DLA  has  had  a  lower  rate  of  protests  at  GAO  than  the  Army,  Navy, 
Air  Force,  or  DoD  (Maser,  Subbotin,  &  Thompson,  2010). 

Procurement  agencies  have  not  been  aggressive  in  implement¬ 
ing  an  executive  order  requiring  agencies  to  create  ADR  systems 
(Nabatchi,  2007).  First,  it  was  a  relatively  new  concept,  so  agencies 
were  not  convinced  that  it  served  their  organizational  interests.  Sec¬ 
ond,  few  internal  pressures  existed  to  use  it,  and  external  actors  were 
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not  clamoring  for  it.  Third,  agencies  had  little  empirical  evidence  of 
its  merits.  However,  the  concept  is  no  longer  new.  The  likelihood 
of  increasing  numbers  of  protests  that  forestall  execution  makes 
agency-level  reviews  more  efficacious.  The  success  of  DLA’s  and 
GAO’s  processes  testifies  to  the  merits  of  ADR  on  principle.  From 
an  incentive  perspective,  requiring  agency-level  reviews  would  give 
agencies  added  incentives  to  document  their  decisions,  and  thereby 
improve  disclosure,  especially  if  their  responses  become  part  of  the 
record  before  GAO.  Whether  it  will  work  with  source  selections  more 
complex  than  those  at  agencies  like  DLA  merits  testing. 

Principle  No.  2:  Provide  Loop-Backs  to  Interest-Based 
Negotiation 

The  bid  protest  system  encourages  parties  to  do  this  by  allowing 
agencies  to  take  corrective  actions.  At  any  time,  including  during 
GAO’s  proceedings,  an  agency  can  respond  affirmatively  to  the 
claims  made  by  an  offeror,  who  then  withdraws  the  protest.  GAO 
has  a  specialized  form  of  ADR  nested  within  it— predictive-dispute 
resolution— where  GAO  predicts  the  outcome  of  its  decision  making 
based  on  a  preliminary  analysis. 

This  has  resolved  a  high  percentage  of  protests  by  parties  who 
have  participated  in  it. 

Incentives  for  offerors  to  negotiate  seriously  could  be  imple¬ 
mented.  If  a  rejected  offeror  lodges  multiple  protests  with  one  or 
more  agencies,  who  all  conclude  that  the  protests  have  no  merit, 
and  if  GAO  subsequently  agrees,  then  after  some  number  of  pro¬ 
tests,  such  as  three  in  3  years,  GAO  could  be  empowered  to  require 
the  rejected  offeror  to  begin  compensating  the  agencies  for  their 
costs  associated  with  responding  to  the  protests.  Failing  Con¬ 
gress  authorizing  GAO  to  do  that,  GAO  could  begin  documenting 
repeatedly  frivolous  protest  behavior  as  part  of  past  performance 
data  that  agencies  consider  in  making  awards.  This  merely  makes 
transparent  and  systematic  something  contractors  already  believe 
transpires  in  obscurity  and  inconsistently. 

Principle  No.  3:  Provide  Law-Cost  Rights  and  Power  Backups 

If  interests-based  negotiations  do  not  yield  a  resolution  accept¬ 
able  to  both  parties,  they  require  access  to  a  determinative  process. 
This,  in  effect,  is  what  GAO’s  bid  protest  process  represents  relative 
to  the  COFC.  The  process  looks  more  like  nonbinding  arbitration 
than  mediation,  although  it  does  not  fit  neatly  into  any  traditional 
model  of  ADR.  Arbitration,  chosen  primarily  for  its  finality  and  effi¬ 
ciency,  is  more  about  consistency— within  limits.  Something  akin  to 
a  precedent  can  emerge,  but  not  as  rigorous  a  body  of  precedent 
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and  law  governing  discovery  and  evidence  as  an  Article  3  court 
(Metzger  &  Lyons,  2007). 

GAO  should  monitor  and  be  transparent  about  its  standards  of 
materiality  and  reasonableness  and  the  processes  by  which  they  are 
assured.  A  higher  standard  might  be  appropriate  for  incumbents 
who  protest,  either  in  terms  of  agencies  providing  a  rationale  for 
changing  suppliers  or  in  terms  of  GAO’s  standard  of  the  reason¬ 
ableness  of  an  incumbent’s  claim.  If  an  agency  has  had  experience 
with  an  incumbent  and  still  believes  a  new  contractor  is  preferable, 
GAO  could  afford  greater  deference  to  the  agency.  This  offsets,  in 
part,  the  incumbent’s  informational  advantage  in  the  competition 
and  its  incentives  to  protest  to  extend  its  contract  for  the  period  of 
the  protest  review. 

Principle  No.  4:  Build  in  Consultation  Before,  Feedback  After 

As  a  form  of  consultation  between  agencies  and  rejected 
offerors,  debriefings  remain  key  to  improving  trust  and  mitigating 
protests.  To  promote  disclosure  means,  ironically,  that  agencies 
should  assume  rejected  offerors  will  protest.  Agencies  should  sup¬ 
ply  them  the  same  information  provided  to  the  Source  Selection 
Authority,  which  is  to  say,  the  same  level  of  detail  that  the  agency 
should  provide  in  responding  to  a  protest.  Agencies  should  be  able 
to  explain  to  the  offeror  why  that  offeror  was  not  selected;  if  the 
agency  cannot  do  that  in  a  debriefing,  then  it  will  not  be  able  to 
defend  itself  in  a  protest. 

A  Source  Selection  Joint  Action  Team  in  the  Office  of  the  Sec¬ 
retary  of  Defense  is  looking  at  the  consistency  of  debriefings.  This 
office  or  its  designated  agents  should  collect  data  continuously 
about  the  quality  of  the  debriefings  to  compare  performance  with 
expectations  and,  thereby,  to  continue  to  improve  them.  A  related 
recommendation  is  to  mitigate  the  adversarial  tone  in  debriefings. 
Rather  than  explaining  how  the  rejected  offeror  erred  and  should 
change  so  as  to  help  the  agency,  the  debriefing  ought  to  explain 
what  the  offeror  should  do  better  to  help  itself  (Thompson,  2009, 
p.  165). 

Another  way  to  improve  consistency  is  to  record  debriefings. 
This  might  obviate  the  need  for  attorneys  to  attend  them.  It  also 
obviates  the  need  for  protestors  to  solicit  affidavits  from  everyone 
at  their  companies  who  attends  a  debriefing.  It  also  supports  GAO’s 
job  as  third-party  intervener.  A  contracting  official  can  say  the  same 
thing  in  debriefings  to  two  different  protestors;  depending  on  the 
attitudes  and  interests  of  the  protestors,  one  will  find  the  contract¬ 
ing  official  unresponsive  and  the  other  will  not.  If  a  protest  results, 
GAO  can  judge. 
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Finally,  GAO  reports  to  Congress  annually  on  the  number  of 
bid  protests  and  their  resolution,  including  agency  responses  to 
GAO  decisions  and  the  number  of  corrective  actions.  To  improve 
the  efficacy  of  its  bid  protest  system,  GAO  should  track  and  report 
more  fully  to  Congress  on  the  outcomes  of  agency  decisions.  If, 
for  example,  an  agency  agrees  to  take  corrective  action— which  is 
generally  viewed  as  a  positive  outcome— by  initiating  a  new  solici¬ 
tation  but  takes  years  to  do  so,  devaluing  the  investment  made  by 
the  contract  winner  and  putting  other  offerors  on  hold,  the  result 
is  not  necessarily  salutary.  Congress  should  have  better  feedback 
on  what  works  and  what  does  not. 

Principle  No.  5:  Provide  Necessary  Motivation,  Skills,  and 
Resources 

To  make  agency-level  reviews  more  credible,  agencies  should 
use  staff  trained  in  negotiation  and  mediation,  preferably  using 
parties  different  from  those  engaged  in  the  initial  decisions  (Troff, 
2005,  pp.  145-149).  The  same  idea  applies  to  peer  reviews  of  source 
selections,  formally  instituted  in  2008.  Peer  reviewers  should  be 
trained  to  conduct  a  peer  review,  which  is  different  than  conduct¬ 
ing  a  source  selection;  and  their  involvement  should  become  part  of 
their  performance  evaluation  and  tracked,  which  does  not  happen 
systematically.  Finally,  if  certification  becomes  the  coin  of  the  realm 
in  the  government’s  Defense  Acquisition  Workforce  (Fast,  2009), 
contractors  could  be  required  to  have  staff  whose  understanding 
of  the  source  selection  process  is  certified,  as  well. 

Another  skill  to  improve  consistency  in  source  selections  would 
be  for  agencies  to  invest  in  managing  risk  in  the  way  companies  do. 
DoD  has  developed  a  methodology  for  risk  management  that  lodges 
primary  responsibility  with  the  program  manager,  and  explained  it 
in  Risk  Management  Guide  for  DoD  Acquisition  (DoD,  2006).  It 
focuses  on  three  risks:  performance,  cost,  and  schedule.  The  term 
“source  selection”  never  appears.  The  term  “contracting  officer” 
appears  twice.  While  it  purports  to  apply  to  the  entire  acquisition 
process,  it  focuses  on  contracting  and  contract  execution  after 
source  selection. 

Yet  source  selection  is  all  about  managing  risks.  Technical  risk 
concerns  the  ability  of  the  product  to  perform  to  specifications. 
Financial  risk  concerns  the  ability  to  deliver  the  product  or  service 
on  budget.  Sustainment  risk  concerns  the  ability  to  maintain  the 
product  within  budget.  Congressional  risk  concerns  political  sup¬ 
port  for  or  interest  in  the  product  or  service.  Appropriation  risk 
concerns  the  ability  of  Congress  to  continue  funding  the  product 
or  service.  Reputational  risk  concerns  the  ability  of  the  agency  to 
execute  a  successful  source  selection.  Bid  protest  risk  concerns 
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the  likelihood  of  drawing  a  protest.  Some  risks  are  internal  to  the 
agency;  others  are  external  (Rogerson,  1994).  Some  are  tangible; 
others  less  so. 

Companies  face  an  analogous  set  of  risks  to  their  success  in 
bidding,  although  they  perceive  risk  differently  (Frick,  2010).  Many 
have  a  chief  risk  officer  (CRO);  agencies  do  not.  Some  believe 
the  Source  Selection  Advisory  Committee  identifies  risks  for  the 
Source  Selection  Authority,  who  plays  the  CRO.  Other  current  and 
former  contracting  officials  see  no  one  responsible  for  identifying 
the  complete  range  and  extent  of  risks  in  source  selections  and 
apprising  decision  makers  of  their  options  and  tradeoffs  in  manag¬ 
ing  them.  While  methods  for  adapting  corporate  risk  management 
processes  to  acquisitions  have  been  developed  (Rice,  2010),  sys¬ 
tematically  assigning  responsibility  and  authority  for  using  them 
has  not.  Adopting  common  methods  and  structural  arrangements 
to  manage  risk  could  reduce  perceptions  of  bias  and  inconsistency. 

Conclusions 

Given  the  inevitability  of  human  error,  good  source  selection 
processes  will  not  eliminate  protests.  As  an  agency  official  put  it: 
"The  question  is  one  of  reasonableness.  If  one  puts  in  place  pro¬ 
cesses  to  ‘perfect’  the  source  selection  so  as  to  minimize  protests, 
you’ll  create  delays  in  producing  the  award  that  may  exceed  the 
delays  caused  by  the  protest  process.”  Once  an  offeror  decides  to 
protest,  in  the  words  of  a  bid  protest  attorney:  “Alternative  dispute 
resolution  works.”  It  provides  a  valuable,  inexpensive  way  of  resolv¬ 
ing  disputes,  especially  compared  to  the  judicial  process.  Another 
said,  “It  limits  the  time  you  are  spending  a  client’s  money.  GAO  helps 
the  sanity  of  the  acquisitions  community.” 

Principles  of  ADR  and  dispute  systems  design  can  prevent 
conflict  spirals  and  mitigate  those  that  begin.  Strengthening 
agency-level  review  can  reconcile  the  interests  of  disputants  at 
low  cost.  Peer  reviews  and  greater  disclosure  through  thorough 
debriefings  can  increase  the  parties’  satisfaction  with  the  process 
and  its  outcomes.  Treating  risk  management  in  source  selection 
as  systematically  as  it  is  treated  in  contract  execution  can  improve 
consistency  in  decision  making  and,  thereby,  relationships  among 
the  parties,  including  not  only  agencies  and  contractors,  but  also 
Congress  and  other  stakeholders.  Greater  transparency  in  GAO’s 
standards  can  promote  GAO’s  multiple  missions,  which  in  turn  can 
produce  more  durable  resolutions. 
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MAXIMIZING  FEDERAL 
IT  DOLLARS: 

A  CONNECTION  BETWEEN 
IT  INVESTMENTS  AND 
ORGANIZATIONAL 
PERFORMANCE 


f  BG  Ennis  C.  “Jim"  Whitehead,  USAR  (Ret.), 
Shah  ram  Sarkani,  and  Thomas  A.  Mazzuchi 

Evaluating  how  best  to  invest  government  information  tech¬ 
nology  (IT)  dollars  means  making  choices.  Should  agencies 
strengthen  infrastructure  with  energy-efficient  servers  and 
increased  network  bandwidth,  purchase  software  to  cut  costs, 
increase  collaboration,  or  invest  more  to  meet  stakeholders’ 
future  needs?  Is  there  a  connection  between  the  way  agencies 
invest  IT  dollars  and  successful  mission  accomplishment?  In 
this  article,  the  authors  show  a  connection  between  IT  invest¬ 
ment  allocations  and  organizational  performance  in  federal 
government  agencies,  and  demonstrate  how  higher  performing 
agencies  invest  differently  in  IT  than  lower  performing  agencies. 
Federal  managers  can  compare  their  organization’s  IT  invest¬ 
ment  portfolio  with  high-performing  agencies  and  compare 
their  investment  allocations  with  other  federal  organizations 
with  similar  missions  to  determine  optimum  IT  investment 
allocations  for  their  agencies. 


Keywords:  Information  Technology  (IT),  IT  Investment 
Portfolio,  Organizational  Performance,  Program 
Assessment  Rating  Tool  (PART),  Federal  Budget 
Process 
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"The  federal  government  has  largely  missed  out  on  the  transfor¬ 
mation  in  the  use  of  IT  due  to  poor  management  of  its  technology 
investments.  Government  IT  projects  all  too  often  cost  millions  of 
dollars  more  than  they  should,  take  years  longer  than  necessary  to 
deploy,  and  deliver  technologies  that  are  obsolete  by  the  time  they 
are  completed. " 

—Vivek  Kundra 

U.S.  Chief  Information  Officers  Council,  July  I,  2010 

So  testified  U.S.  Chief  Information  Officer  Vivek  Kundra  before 
the  House  Committee  on  Oversight  and  Government  Reform,  July 
1,  2010.  The  federal  information  technology  (IT)  budget  stands 
at  $87.9  billion  for  2010,  with  the  Department  of  Defense  (DoD) 
budget  consuming  $35.7  billion  of  this  total  (Office  of  Manage¬ 
ment  and  Budget  [OMB],  2010).  The  government  faces  significant 
pressure  to  improve  agency  performance  and  reduce  IT  costs,  but 
challenges  and  questions  emerge.  Is  there  a  connection  between  IT 
investments  and  agency  performance?  If  so,  where  should  IT  funds 
be  spent  to  be  most  effective— on  improved  IT  infrastructure  with 
more  energy-efficient  servers  and  increased  network  bandwidth, 
or  on  new  software  to  cut  costs  and  increase  productivity?  Would 
IT  dollars  be  most  effectively  used  to  increase  collaboration  and 
management  control,  or  should  agencies  forecast  their  stakehold¬ 
ers’  future  needs  and  invest  more  to  meet  those  requirements? 

With  each  new  administration,  laws  and  programs  are  created  to 
improve  agency  performance  through  improved  IT  productivity.  In 
1996,  the  Clinger-Cohen  Act  adopted  a  private  sector  IT  investment 
approach  of  performance  and  results-based  management;  in  2002, 
e-Government  initiatives  were  begun  to  bring  government  services 
online;  and  today,  agencies  are  looking  at  Information  and  Commu¬ 
nications  Technologies  (ICTs)  to  improve  services  to  constituents. 
If  we  could  find  evidence  that  certain  categories  of  IT  investments 
were  associated  with  higher  organizational  performance,  perhaps 
we  could  spend  IT  budget  resources  more  effectively. 

Sinan  Aral  and  Peter  Weill  at  the  Massachusetts  Institute  of  Tech¬ 
nology  Center  for  Information  Systems  Research  (MIT  CISR)  found 
that  corporate  performance  was  affected  by  IT  asset  allocation 
(Aral  &  Weill,  2007).  Although  federal  government  requirements 
and  measures  of  performance  differ  significantly  from  those  of  the 
private  sector,  with  agencies  focusing  on  mission  accomplishment 
and  responding  to  political  conditions  more  than  market  conditions 
(Holmes,  2001;  Ostroff,  2006),  this  research  examines  a  similar  con¬ 
nection  between  federal  IT  investments  and  agency  performance. 

Studies  of  IT  investments  and  organizational  performance  have 
been  conducted  in  the  private  sector  for  two  decades,  but  research 
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has  not  been  applied  to  the  federal  sector.  The  authors  of  this  study 
examined  federal  government  agency  IT  portfolio  investments  for 
30  agencies,  as  provided  annually  to  the  OMB  on  their  Exhibit  53 
data  submissions,  and  divided  federal  agency  IT  investments  into 
four  categories:  Innovation,  Management  Support,  Process  Auto¬ 
mation,  and  Infrastructure.  They  statistically  compared  agency 
program  performance,  as  determined  from  the  OMB’s  Program 
Assessment  Rating  Tool  (PART),  with  IT  investment  allocations  and 
sought  to  answer  the  following  question: 

Do  higher  performing  federal  government  agencies  invest  IT 
assets  differently  than  lower  performing  agencies? 

Evidence  pointed  to  a  significant  difference  in  how  higher  per¬ 
forming  and  lower  performing  agencies  invest  their  IT  budgets.  A 
causal  effect  between  IT  investment  and  agency  performance  could 
not  be  statistically  proven,  but  significant  differences  were  found. 
This  research  provides  a  new  perspective  on  IT  investment  alloca¬ 
tion  and  suggests  a  technique  by  which  IT  investments  in  the  federal 
government  can  be  evaluated  and  adjusted  to  achieve  agency  goals. 

Definitions 

IT  Assets  and  Investments 

The  U.S.  Government  Accountability  Office,  formerly  the  Gen¬ 
eral  Accounting  Office  (GAO),  adopted  a  definition  of  IT  assets  in  its 
2000  report  on  Information  Technology  Investment  Management: 
“computers,  ancillary  equipment,  software,  firmware  and  simi¬ 
lar  procedures,  services  (including  support  services),  and  related 
resources  used  by  an  organization  to  accomplish  a  function”  (GAO, 
2000).  IT  investments,  the  report  cited,  are  “the  actual  expenditure 
of  resources  on  selected  information  technology  or  IT-related  ini¬ 
tiatives  with  the  expectation  that  the  benefits  from  the  expenditure 
exceed  the  value  of  the  resources"  (emphasis  added)  (GAO,  2000). 
IT  investments  are  expected  to  create  value  for  any  organization, 
private  or  public  sector,  and  at  least  for  the  long  term,  to  return 
more  than  their  costs. 

Organizational  Performance 

Organizational  achievement  for  federal  government  agencies 
includes  continuous  improvement  on  mission  goals  (Popovich, 
1998);  cost-effective  program  delivery,  accountability  to  taxpayers, 
improved  productivity,  and  human  resources  strength,  including 
quality  of  the  workforce  and  employee  satisfaction  (Kaplan  &  Nor¬ 
ton,  2005;  Keyes,  2005);  and  the  approval  of  political  stakeholders 
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TABLE  1.  PRIVATE  SECTOR  VS.  FEDERAL  AGENCY 
ORGANIZATIONAL  PERFORMANCE 


Private  Sector 

Federal  Agencies 

Strategic  Goal 

Competitiveness 

Mission  effectiveness 

Financial  Goal 

Profit,  growth,  market 

share 

Cost  reduction,  efficiency 

Values 

Innovation,  creativity, 

Accountability  to  public, 

good  will,  recognition 

integrity,  fairness 

Customer  satisfaction 

Stakeholder  satisfaction 

Desired  Outcomes 

(customer  pays  for 

(stakeholder  may  not  pay 

product/service) 

proportionally  for  service) 

Stakeholders 

Stockholders,  owners, 

market 

Taxpayers;  legislative,  executive, 
and  judicial  branches 

Budget  Priorities 
Defined  by: 

Customer  demand 

Leadership,  legislators,  planners 

Justification  for 

Secrecy 

Protection  of 

intellectual  capital, 
proprietary  knowledge 

National  security 

Growth  rate,  earnings, 

Best  management  practices, 

Key  Success 

market  share 

legislative  compliance 

Factors 

Uniqueness 

Sameness,  economies  of  scale 

Advanced  technology 

Standardized  technology 

Note.  Adapted  from  “Differing  Measures  for  Private  Sector  and  Federal  Agency  Organizational 

Performance,"  by  P.  Arveson,  1999. 

and  the  public  perception  that  they  are  “doing  the  right  thing” 
(Holmes,  2001).  Since  the  Clinger-Cohen  Act  attempted  to  replicate 
IT  investment  portfolio  success  in  the  private  sector  (Van  Over, 
2009),  and  earlier  IT  investment  portfolio  research  is  based  on  the 
private  sector,  it  is  instructive  to  examine  some  of  the  differences 
between  public  and  private  sector  performance  measures.  Table  1 
is  an  adaptation  of  a  chart  by  Paul  Arveson  (1999),  customized  for 
federal  government  agencies. 

Background 

IT  portfolio  management  is  rooted  in  Markowitz’  Modern  Portfo¬ 
lio  Theory  for  investments,  where  diversification  of  financial  assets 
(stocks,  bonds,  and  cash)  is  balanced  by  expected  returns  and  risk 
(Markowitz,  1952).  Warren  McFarlan  (1981)  used  the  work  of  Mar¬ 
kowitz  to  apply  the  principles  of  investment  and  risk  to  information 
systems  investment,  noting  that  IT  project  and  portfolio  risk  alone 
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was  neither  positive  or  negative,  but  must  be  seen  in  context  to  the 
degree  of  risk  and  potential  reward.  IT  portfolio  risk  requires  exam¬ 
ining  current  and  future  projects,  constantly  evaluating  projects  to 
determine  effectiveness,  and  investing  and  divesting  as  necessary 
(Van  Over,  2009). 

Several  IT  investment  portfolio  allocation  methods  have  been 
proposed  over  the  years.  Glen  Peters  (1988)  and  John  Ward  (1990) 
both  divided  IT  investments  into  functional  categories.  Bryan  Mai- 
zlish  (2005)  furthered  this  concept  with  an  IT  asset  portfolio  system, 
which  allocated  investments  into  Infrastructure,  Process,  Informa¬ 
tion/Data,  and  Human  Capital  investments.  He  noted  that  in  the 
federal  government  extra  challenges  are  inherent  to  successfully 
designing  an  IT  asset  portfolio:  Few  common  standards  or  historical 
bases  are  available  for  evaluating  IT  investments;  IT  investments  are 
difficult  to  retire  without  other  agency  systems  and  databases  being 
affected;  investments  and  their  component  interdependencies  must 
be  monitored,  adjusted,  and  disposed  of  for  their  full  life  cycle;  and 
IT  projects  must  be  evaluated  not  only  for  performance  goals,  risk 
management,  and  life-cycle  cost  formulation,  but  also  for  security 
and  privacy,  and  support  of  the  President’s  management  agenda. 

G.  David  Garson  (2003)  noted  that,  traditionally  IT  invest¬ 
ments  were  made  on  a  project  basis— a  conservative  policy  with 
riskier  projects  often  not  funded.  Portfolio  management  allows 
higher  risk/higher  payoff  projects  to  be  balanced  with  less  risky/ 
lower  payoff  projects,  justifying  some  large  nontraditional  sys¬ 
tems,  including  e-Government  initiatives.  An  awareness  of  the  IT 
investment  process  must  be  built  into  the  agency  so  that  formal 
processes  of  IT  evaluation  are  adopted,  and  the  agency  moves  from 
a  project-centric  base  to  a  portfolio  approach,  evaluating  invest¬ 
ments  according  to  their  support  of  the  agency’s  overall  mission. 
High-risk,  low-value,  or  obsolete  IT  investments  are  evaluated  and 
may  be  de-selected  from  the  portfolio,  and  through  benchmark¬ 
ing  IT  investments  to  successful  organizational  investments,  better 
technologies  can  be  chosen. 

Researchers  from  the  MIT  CISR  have  done  extensive  research 
on  private  company  and  industry  IT  investment  portfolios.  Using 
survey  data  from  chief  information  officers  or  their  representatives 
from  1,508  companies  in  60  countries  over  a  10-year  period,  and 
statistically  controlling  for  industry,  firm  size,  advertising  expense, 
and  research  and  development,  they  divided  IT  expenditures  into 
four  asset  classes:  Infrastructure,  Transactional,  Informational,  and 
Strategic.  From  this  data,  they  established  patterns  within  industries, 
evaluated  business  strategies,  and  defined  measures  companies 
could  take  to  evaluate  their  current  IT  expenditures  and  improve 
organizational  performance. 
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Weill  and  Aral  (2003),  Weill  and  Ross  (2004),  Aral  and  Weill 
(2007),  and  Weill  and  Aral  (2008)  found  that: 

1.  Strategic  investments,  which  are  designed  to  gain  com¬ 
petitive  advantage  in  the  marketplace  and  to  develop 
new  business  products  and  services,  on  average  con¬ 
sumed  13  percent  of  private  sector  IT  investments  in 
2007.  These  investments  are  higher  risk,  have  a  longer 
lead-time,  and  often  involve  very  new  technologies. 

The  failure  rate  for  strategic  IT  investments  can  be 
as  high  as  50  percent,  but  their  successes  could  put 
a  company  several  years  ahead  of  competitors.  Weill 
noted  that  strategic  investments  in  the  small  sample  of 
public  sector  organizations  (charities,  private  schools, 
local  government)  he  studied  led  to  greater  innovation, 
increased  interaction  with  customers,  enabled  major 
changes,  and  easier  facilitation  within  the  organization. 

2.  Informational  investments,  which  provide  internal  infor¬ 
mation  (e.g.,  for  accounting,  and  communication)  are 
designed  to  reduce  costs  and  add  potential  profitability 
improvements  in  the  future.  In  2007,  they  consumed  on 
average  13  percent  of  annual  firm  IT  investment. 

3.  Transactional  investments,  which  often  automate  exist¬ 
ing  operations,  may  result  in  immediate  cost  reductions. 

In  2007,  the  average  private  sector  organization  studied 
by  Weill  apportioned  27  percent  of  annual  IT  investment 
to  transactional  investments.  Private  sector  companies 
that  invest  more  heavily  in  transactional  systems  than 
their  competitors  had  higher  productivity  (sales  per 
dollar  of  assets)  and  lower  costs.  They  estimate  trans¬ 
actional  investments  return  25-40  percent  per  IT  dollar 
invested. 

4.  For  Infrastructure  costs,  the  objective  is  to  either  reduce 
IT  costs  via  consolidation  or  to  establish  a  flexible, 
reusable  base  for  future  business  needs.  Infrastructure 
investments  typically  have  high  initial  costs  and  lower 
short-term  profitability,  but  higher  operational  perfor¬ 
mance  and  profitability  over  the  long  run.  In  2007,  the 
average  firm  in  the  CISR  study  allocated  47  percent  of 
IT  investment  to  infrastructure. 

One  important  consideration  for  government  organizations  is 
their  high  legacy  system  costs.  When  Weill,  Woerner,  and  Rubin 
(2008)  looked  at  sustaining  IT  investments,  which  maintain  and 
update  current  systems,  and  new  investments,  which  encompass 
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major  initiatives  and  changes  to  systems,  they  found  that  in  2007 
the  average  firm  spent  66  percent  of  IT  investments  on  sustaining 
investments,  and  one-third  of  the  firms  studied  by  the  MIT  CISR 
spent  over  75  percent  of  their  IT  dollars  to  run  existing  systems. 
With  so  much  of  IT  assets  spent  supporting  existing  complex  and 
often  redundant  systems,  portfolio  assets  are  not  freed  up  to  sup¬ 
port  new  IT  systems.  Weill  found  that  organizations  allocating  more 
to  new  investments  rather  than  sustaining  investments  had  greater 
revenue  growth,  and  by  that  measure,  were  higher  performing  in 
their  industry.  Top  performers  in  each  industry  spent  4  percent 
more  on  IT,  but  had  similar  portfolios  (Weill  &  Aral,  2004). 

Research  Data  and  Methods 

Our  study  adapted  private  sector  research  methods  relating 
organizational  performance  and  IT  investment  portfolios  to  U.S. 
Government  agencies.  Much  of  our  comparative  resource  informa¬ 
tion  came  from  studies  by  the  MIT  CISR,  but  because  of  differences 
between  the  public  and  private  sector,  we  adapted  our  categories 
to  public  sector  requirements. 

Agency  Selection 

Federal  government  agencies  in  this  study  were  selected  for 
comparative  purposes  if  both  publicly  available  performance  data 
and  IT  investment  data  were  available.  Thirty  agencies  were  evalu¬ 
ated,  as  shown  in  Table  2.  The  10  highest  performing  agencies 
were  compared  to  the  10  lowest  performing  agencies  in  terms  of 
their  IT  investment  allocations.  Agencies  with  similar  mission  focus 
were  also  disaggregated  and  compared  within  each  focus  area  to 
develop  trends. 

Data  for  Agency  Performance 

To  determine  agency  performance,  we  looked  for  publicly  avail¬ 
able,  objective  measures  by  which  federal  government  agency 
performance  could  be  ranked.  Comparative  data  are  limited,  and 
agencies  are  reluctant  to  be  compared  with  one  another:  They  per¬ 
ceive  themselves  as  unique  in  mission  and  resources— both  of  which 
are  dictated  by  Congress— with  changing  political  environments  and 
politically  determined  budgets.  Comparisons  could  only  be  found 
for  employee  satisfaction  (Brewer,  2000),  management  excellence 
(Office  of  Personnel  Management,  2008),  and  website  technology 
effectiveness  (West  &  Lu,  2009). 

The  Government  Performance  and  Results  Act  of  1993  estab¬ 
lished  a  framework  for  department  and  agency  performance 
reporting,  designed  to  assist  federal  organizations  in  improving 
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TABLE  2.  AGENCIES  EVALUATED  FOR  THIS  STUDY 

Thirty  Agencies  Evaluated  in  this  Study 


Department  of  Agriculture 
Department  of  the  Air  Force 
Department  of  the  Army 
Department  of  Commerce 
Department  of  Defense  —  Other 
Department  of  Education 
Department  of  Energy 
Department  of  Health  and 
Human  Services 

Department  of  Homeland  Security 
Department  of  Housing  and  Urban 
Development 
Department  of  the  Interior 
Department  of  Justice 
Department  of  Labor 
Department  of  the  Navy 
Department  of  State 
Department  of  the  Treasury 


Department  of  Transportation 
Department  of  Veterans  Affairs 
Environmental  Protection  Agency 
General  Services  Administration 
National  Aeronautics  and 
Space  Administration 
National  Archives  and 
Records  Administration 
National  Science  Foundation 
Nuclear  Regulatory  Commission 
Office  of  Personnel  Management 
Small  Business  Administration 
Smithsonian  Institution 
Social  Security  Administration 
US  Agency  for  International 
Development 

U.S.  Army  Corps  of  Engineers  — 
Civil  Works 


program  performance.  Through  2008,  the  OMB  had  assessed  the 
performance  of  1,017  federal  government  programs,  representing 
98  percent  of  the  federal  budget,  using  the  Program  Assessment 
Rating  Tool  (PART)  (OMB,  2009a). 

The  PART  is  an  annual  agency  report  that  evaluates  each 
agency’s  key  programs  in  four  key  areas:  purpose,  performance, 
measurement,  and  results  across  a  common  matrix.  The  PART  ques¬ 
tionnaire  asks  25  questions  specific  to  the  category  of  each  program, 
and  must  be  documented  with  data.  Agencies  have  some  flexibility 
in  choosing  methods  of  evaluation,  which  must  be  approved  by 
the  OMB.  Four  areas  are  assessed:  purpose  and  design,  strategic 
planning,  management  and  results,  and  accountability.  A  numeric 
score  is  derived  from  each  of  the  four  areas  of  assessment,  ranging 
from  0-100,  with  100  being  the  best.  These  numbers  determine  one 
of  four  qualitative  ratings:  Effective  (85-100),  Moderately  Effective 
(70-84),  Adequate  (50-69),  or  Ineffective  (0-49).  If  a  program  is 
not  measured  by  acceptable  performance  measures  or  does  not 
yet  have  performance  data,  a  rating  of  Results  Not  Demonstrated 
is  given.  Programs  may  be  reassessed  after  corrective  actions  are 
completed  to  improve  their  ratings  (OMB,  2009a;  OMB,  2009b). 

By  design,  the  PART  has  historic  agency  data  on  listed  programs 
that  reflect  the  latest  evaluations,  including  any  reassessments.  It 
was  initiated  in  2004  with  approximately  one-fifth  of  agency  pro- 
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grams  rated  each  year  until  the  PART  reflected  all  1,017  programs 
in  2008.  The  2008  reporting  year  was  selected  to  evaluate  the 
maximum  number  of  programs,  and  the  2-year  mean  was  used  to 
establish  weighted  scores  for  program  size  in  each  agency.  With 
these  numbers,  agencies  could  be  ranked  as  to  the  effectiveness  of 
the  programs  they  self-described  as  most  important  to  their  agency 
mission,  using  cluster  analysis  in  performance  categories  of  “highest 
performing”  (top  third  of  grouped  agencies)  and  “lowest  perform¬ 
ing”  (bottom  third  of  grouped  agencies). 

Only  agency  information  that  was  publicly  available  at  the  time 
of  this  study  was  used.  Intelligence  agencies  participate  in  these 
evaluations,  but  their  results  are  not  publicly  available  and  not 
included  in  this  report.  The  methods  and  results  appear  to  be  con¬ 
sistent  throughout  different  organizations,  however,  and  should  be 
internally  applicable  to  these  agencies  as  well. 

Data  for  IT  Investment  Allocations 

The  Clinger-Cohen  Act  of  1996  put  the  OMB  in  charge  of  improv¬ 
ing  the  productivity,  efficiency,  and  effectiveness  of  federal  agencies 
by  linking  planning  and  investment  strategies  and  IT  portfolio  man¬ 
agement  to  the  federal  budget  process.  Each  agency  is  required  to 
create  capital  IT  asset  portfolios  and  review  them  to  determine  if  a 
project  is  still  attainable  and  has  a  high  benefit/cost  ratio  compared 
with  other  investments  in  the  portfolio.  The  agency  annual  IT  invest¬ 
ment  portfolio,  known  as  OMB  Exhibit  53,  is  a  reporting  mechanism 
for  agencies  to  evaluate  all  IT  projects  and  ensure  that  they  are 
well-planned  and  meet  cost,  schedule,  and  performance  goals 
planned  for  the  investment.  It  has  six  major  categories  of  IT  invest¬ 
ments:  Mission  Area  Support;  Infrastructure,  Office  Automation,  and 
Telecommunications;  Enterprise  Architecture  and  Planning;  Grants 
Management  Systems;  Grants  to  State  and  Local  IT  Investments; 
and  National  Security  Systems.  Exhibit  53  is  used  by  each  agency 
to  report  the  information  in  its  annual  IT  investment  portfolio  for 
both  major  and  nonmajor  programs  to  the  OMB,  and  is  published 
as  part  of  the  federal  budget.  It  is  designed  to  assist  agencies  in 
selecting  investments  to  improve  the  management  of  IT  programs, 
understand  the  amount  spent  on  IT  modernization  and  support  of 
legacy  systems,  and  encourage  interagency  cooperation  to  elimi¬ 
nate  redundant  and  nonproductive  IT  investments.  The  purpose  of 
Exhibit  53  is  to  encourage  agencies  to  focus  IT  spending  on  high- 
priority  modernization  initiatives;  to  manage  major  IT  investments 
within  10  percent  of  cost,  schedule,  and  performance  objectives; 
and  to  protect  the  security  of  information  systems  (OMB,  2008). 

Data  for  this  research  were  taken  from  the  2008  Exhibit  53. 
Over  7,200  IT  investments  were  listed  for  the  2008-2009  bud- 
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FIGURE  1.  AGENCY  IT  INVESTMENT  DISAGGREGATION  PROCESS 


L  Innovation 

Agency  IT  Using  project  descriptions  and  codes, 

Management  Support 

Investments  investments  are  divided  into  four  categories 

W  Process  Automation 

f  Infrastructure 

get  years  for  the  30  departments  and  agencies  used  in  the  study 
(OMB,  2009c).  Again,  the  study  used  the  mean  of  2  reporting  years 
to  better  accommodate  any  spikes  in  one  type  of  investment  in  a 
given  year.  This  study  allocated  over  7,200  17-digit  coded  invest¬ 
ments  in  a  total  of  30  federal  government  agencies,  based  on 
project  descriptions  and  codes,  according  to  the  four  investment 
allocation  categories  of  Innovation,  Management  Support,  Process 
Automation,  and  Infrastructure.  These  categories  are  similar  to  the 
four  categories  described  in  Weill  and  Aral  (2003):  Strategic,  Infor¬ 
mational,  Transactional,  and  Infrastructure,  but  adapted  to  better 
describe  Exhibit  53  categories.  Total  IT  investments  were  disaggre¬ 
gated  into  these  four  IT  investment  categories,  as  shown  in  Figure  1. 

1.  Innovation  investments  include  those  investments  that 
provide  a  new  service  or  major  innovation  that  impacts 
external  stakeholders.  Examples  include  the  e-Grants 
portal,  giving  citizens  one  central  location  from  which  to 
access  federal  government  grant  information;  the  Veter¬ 
ans  Administration  e-Gov  Benefits  program,  providing 
a  single  point  of  access  for  citizens  to  locate  and  deter¬ 
mine  potential  eligibility  for  government  benefits;  and 
the  Army’s  Force  XXI  Battle  Command,  Brigade  and 
Below  information  system,  a  new  graphical  information 
system  significantly  improving  battlefield  awareness  for 
commanders. 

2.  Management  Support  investments  are  designed  to  pro¬ 
vide  information  to  employees  to  improve  accounting, 
management,  reporting,  communication,  collaboration, 
or  analysis.  This  would  include  the  DoD’s  Defense  Enter¬ 
prise  Accounting  and  Management  System,  a  financial 
management  system  designed  to  modernize  internal 
accounting  systems,  and  the  Department  of  State’s 
Treaty  Information  Management  System,  which  makes 
treaty  data  more  accessible  to  department  employees. 

3.  Process  Automation  investments  are  used  to  cut  costs  or 
increase  throughput  for  the  same  cost  in  organizational 
operations,  often  through  automating  existing  opera¬ 
tions.  Examples  include  the  National  Institutes  of  Health 
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Electronic  Research  Administration  System,  which 
automates  formerly  paper-based  functions  in  grant 
administration;  the  Department  of  Labor  PeoplePower 
system,  which  integrates  all  human  resources  pro¬ 
cesses  into  one  system;  and  the  Defense  Travel  System, 
which  automates  travel  authorization  and  vouchering— 
previously  a  manual,  labor-intensive  process. 

4.  Infrastructure  investments  are  shared  resources  used  by 
multiple  applications  (e.g.,  servers,  networks,  desktop 
computers,  and  customer  databases),  and  comprise  a 
substantial  proportion  of  IT  investments. 

In  reality,  individual  investments  often  span  two  or  more  of 
these  categories,  and  can  change  over  time  as  Infrastructure  invest¬ 
ments  are  retired  and  Innovation  investments  become  accepted  as 
mainstream. 

Figure  2  illustrates  these  investment  categories  as  interdepen¬ 
dent  Building  Blocks  of  IT.  Infrastructure  is  at  the  base  and  provides 
support  for  all  other  IT  investments.  At  the  next  level,  Process  Auto¬ 
mation  investments,  which  often  automate  existing  procedures  for 
cost-cutting  purposes,  rely  on  a  solid  base  of  Infrastructure,  but  also 
symbiotically  relate  to  both  Management  Support  and  Innovation 
investments.  Management  Support  investments  improve  commu¬ 
nications  and  operations  within  an  agency,  and  rely  both  on  a  solid 
Infrastructure  base  and  Process  Automation  investments.  Innovation 
investments— the  high-risk/high-potential  investments  providing 
new,  strategic  services  for  stakeholders— rely  both  on  Infrastructure 
as  a  shared  base  and  Process  Automation  investments  to  improve 
cost  effectiveness. 


FIGURE  2.  BUILDING  BLOCKS  OF  IT  INVESTMENTS 
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Reliability  and  Limits  of  the  Data 

Data  from  both  the  PART  and  Exhibit  53  were  obtained  from 
official  public  U.S.  Government  reports.  Each  agency’s  report  is 
required  by  law  annually  and  must  be  verified  by  agency  leader¬ 
ship.  Our  statistical  results  were  limited  to  30  federal  agencies,  and 
expanded  research  in  the  future  could  be  pursued  with  the  addition 
of  agency  subgroups.  We  did,  however,  find  trends  that  were  consis¬ 
tent  throughout  each  of  the  federal  government  agency  categorical 
comparisons  we  studied,  and  were  similar  to  trends  Weill  and  Aral 
(2004)  found  in  the  private  sector. 

It  can  be  argued  that  2  years  of  Exhibit  53  data  is  not  enough, 
since  the  effects  of  IT  investments  may  lag  behind  implementation 
and  therefore  need  a  longer  time  frame.  We  agree  that  further 
research  should  examine  the  Exhibit  53  and  PART  data  over  dif¬ 
ferent  time  frames.  Rai,  Patnayakuni,  and  Patnayakuni  (1997)  note 
that  IT  investments  may  have  less  of  a  lag  effect  than  capital  invest¬ 
ments,  however,  due  to  the  accelerated  rate  of  IT  obsolescence. 
They  further  note  that  more  than  80  percent  of  an  organization’s 
IT  expenditures  are  for  current  operations,  and  the  depreciation  of 
new  hardware  further  dilutes  any  investments,  which  would  impact 
any  lag  time  for  current  performance. 

Recurring  complaints  are  that  the  PART  does  not  accurately 
measure  the  focus  of  the  programs,  allow  flexibility  as  to  the  pro¬ 
gram’s  mandates,  or  give  credit  for  any  programs  that  cannot  be 
currently  quantified,  and  are  therefore  awarded  a  “Results  Not 
Demonstrated”  classification  (Gueorguieva  et  al .,  2009).  The  PART 
is  not  a  perfect  interagency  program  evaluation  tool,  but  it  has 
advantages  for  this  study  in  its  public  availability,  verified  data,  con¬ 
sistent  criteria  among  agencies,  multiyear  time  frame,  allowance  for 
rescoring  and  updating  of  programs,  and  coverage  of  98  percent 
of  the  federal  budget  (OMB,  2009d). 

Results 

Do  higher  performing  agencies,  based  on  program  performance, 
invest  in  IT  differently  than  lower  performing  agencies? 

We  first  examined  the  data  to  determine  normality  of  distribu¬ 
tions.  Using  Minitab  15,  we  could  only  show  normality  within  a  95 
percent  confidence  interval  for  Management  Support  investments 
in  both  the  10  highest  performing  agencies  and  10  lowest  perform¬ 
ing  agencies.  As  a  result,  we  chose  to  use  nonparametric  tools  in  an 
attempt  to  prove  the  null  hypothesis— that  higher  performing  federal 
agencies  have  the  same  IT  investment  strategies  as  lower  perform¬ 
ing  agencies.  Using  the  Mann-Whitney  test,  which  assesses  whether 
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two  independent  samples  of  observations  have  equally  large  values, 
we  showed  that  within  a  95  percent  confidence  interval,  Innovation 
investments  are  different  for  the  highest  and  lowest  performing 
agencies.  Since  the  Innovation  investments  are  different,  the  overall 
portfolios  of  investments  are  different.  The  null  hypothesis  is  not 
supported,  and  therefore,  the  10  highest  performing  agencies  invest 
their  IT  assets  differently  than  the  10  lowest  performing  agencies. 

We  could  not  prove  causal  statistical  correlations,  but  we  can 
see  relationships  between  aggregate  performance  and  IT  invest¬ 
ment  categories.  We  first  looked  at  the  IT  investment  breakdown 
for  the  average  of  the  10  highest  performing  and  10  lowest  perform¬ 
ing  agencies,  and  the  average  of  all  agencies  (Table  3).  We  then 
classified  Cost-focused  and  Agility-focused  agencies  according 
to  agency  mission  statements  and  evaluated  their  IT  investment 
allocations  (Table  4). 

Higher  performing  agencies  spent  twice  as  much  as  the  average 
agencies  and  more  than  five  times  that  of  lower  performing  agen¬ 
cies  for  Innovation  investments  and  less  than  average  agencies  for 
Management  Support  and  Process  Automation  systems.  Infrastruc¬ 
ture  investments  were  consistent  across  all  performance  categories. 
Higher  performing  agencies  invested  41  percent  more  on  IT  than 
the  agency  average  and  71  percent  more  than  lower  performing 
agencies  (Table  3). 


TABLE  3.  IT  INVESTMENT  AVERAGES  FOR  AGENCIES  BY 
RANKING8 


Lower 

Higher 

Four  Categories  of 

Performing 

Average  of 

Performing 

IT  Investments'1 

Agencies 

All  Agencies 

Agencies 

Innovation 

3% 

8% 

16% 

Management 

Support 

16% 

13% 

10% 

Process  Automation 

24% 

23% 

18% 

Infrastructure 

57% 

56% 

56% 

IT  Spending  as 
a  Percentage  of 
Overall  Agency 
Budgetc 

1.68% 

2.04% 

2.87% 

aPerformance  determined  by  ranking  of  PART  scores  over  a  2-year  period. 
b Agency  category  distribution  determined  by  Exhibit  53  data. 

cTotal  Agency  Budget  information  for  2008-2009  retrieved  from  Government  Printing 
Office  (GPO)  Access  database:  http://www.gpoaccess.gov/usbudget/ 
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TABLE  4.  IT  INVESTMENT  STRATEGY  BENCHMARKS  FOR  AGENCIES  BY 
COST-FOCUSED  AND  AGILITY-FOCUSED  MISSION3 


Four  Categories  of 
IT  Investments1* 

Average  Agency 

Cost-Focused 

Agility-Focused 

Innovation 

8% 

5% 

17% 

Management  Support 

13% 

16% 

5% 

Process  Automation 

23% 

30% 

21% 

Infrastructure 

56% 

49% 

57% 

IT  Dollars  as  a 

Percentage  of  Overall 

2.04% 

0.62% 

5.89% 

Agency  Budget0 

aMission  determined  from  agency  websites. 
b Agency  category  distribution  determined  by  Exhibit  53  data. 

cTotal  Agency  Budget  information  for  2008-2009  retrieved  from  Government  Printing  Office  (GPO) 
Access  database:  http://www.gpoaccess.gov/usbudget/ 


DoD  agencies— which  include  the  Department  of  the  Air  Force, 
Department  of  the  Army,  Department  of  the  Navy,  and  Depart¬ 
ment  of  Defense-Other— each  independently  ranked  high  in  the 
PART  evaluation.  Their  IT  investment  portfolio  spending  showed 
the  following  investment  allocations:  Innovation  (30  percent),  Man¬ 
agement  Support  (3  percent),  Process  Automation  (12  percent), 
Infrastructure  (55  percent),  and  an  overall  IT  spending  percentage 
of  5.29  percent,  as  compared  to  their  budget.  These  results  are 
consistent  with  other  top-performing  agencies. 

We  next  divided  the  agencies  into  Cost-focused  and  Agility- 
focused,  according  to  the  missions  stated  on  their  agency  websites. 
Cost-focused  agencies  (e.g.,  the  General  Services  Administration  and 
Social  Security  Administration)  are  committed  to  providing  optimum 
value  for  taxpayers,  and  Agility-focused  agencies  (e.g.,  the  Depart¬ 
ment  of  Defense  and  Homeland  Security)  have  a  primary  objective 
to  protect  the  nation  and  react  promptly  to  mission  changes. 

Cost-focused  agencies  spent  less  than  average  agencies  on 
Innovation  and  Infrastructure,  and  more  than  average  on  Manage¬ 
ment  Support  and  Process  Automation.  Agility-focused  agencies 
spent  twice  as  much  as  average  agencies  on  Innovation,  less  on 
Management  Support  and  Process  Automation,  and  the  same  on 
Infrastructure  (Table  4). 

These  results  are  consistent  with  those  found  in  private  sector 
studies.  Weill  and  Aral  (2003)  noted  that  cost-focused  firms  spent 
less  than  average  on  Strategic  (similar  to  Innovation)  investments, 
Informational  (similar  to  Management  Support)  and  Infrastructure,  and 
significantly  more  on  Transactional  (Process  Automation)  investments. 
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Agility-focused  firms  spent  more  on  Strategic,  less  on  Informational 
and  Transactional,  and  slightly  more  on  Infrastructure  investments. 

Discussion 

This  research  proves  that  federal  government  agencies  that  are 
most  successful  in  program  performance  have  a  different  IT  port¬ 
folio  of  investments  than  those  of  less  successful  agencies.  Higher 
performing  agencies  invest  more  in  Innovation  and  less  in  Manage¬ 
ment  Support  and  Process  Automation  as  a  percentage  of  their  total 
portfolios.  This  result  may  be  because  Innovation  investments  can 
set  the  stage  for  new  services  or  major  agency  improvements,  or 
signal  that  improved  IT  governance  has  allowed  more  of  an  agency 
focus  on  modernization.  Higher  performing  agencies  may  not  place 
top  priorities  on  reducing  costs,  and  therefore  not  channel  their 
resources  into  Management  Support  and  Process  Automation. 
Higher  performing  agencies  also  invest  more  in  IT  as  a  percent¬ 
age  of  their  budgets  than  lower  performing  or  average  agencies, 
possibly  indicating  a  greater  management  focus  on  IT  as  a  way  to 
improve  agency  performance. 

Our  study  also  found  that  Cost-focused  agencies  invest  higher 
than  average  in  Process  Automation,  lower  than  average  in  Inno¬ 
vation  and  Infrastructure,  and  lower  in  IT  as  a  percentage  of  their 
overall  budget.  This  is  completely  consistent  with  private  sector 
results  found  in  the  research  of  Weill  and  Aral  (2003).  Process  Auto¬ 
mation  investments  bring  immediate  cost  savings  while  Innovation 
and  Infrastructure  investments  increase  costs  in  the  short  term  and 
may  never  lead  to  lower  costs  in  the  long  term. 

Finally,  this  study  also  showed  that  Agility-focused  agencies,  like 
DoD  and  all  the  Services,  have  higher  than  average  investments  in 
Innovation,  lower  than  average  investments  in  Management  Sup¬ 
port,  and  higher  than  average  IT  investments  as  a  percentage  of 
their  overall  budget.  Again,  these  results  are  very  consistent  with 
the  private  sector  results  found  in  the  MIT  CISR  research  (Weill  & 
Aral,  2003).  Agility-focused  federal  agencies  and  private  firms  will 
invest  more  in  Innovation  IT  that  brings  new  services  to  stakehold¬ 
ers  and  less  on  Management  Support  that  brings  them  future  cost 
savings  and  future  profitability.  For  DoD  agencies,  it  may  simply  be 
that  the  high  spending  (30  percent)  on  Innovation  in  the  IT  portfolio 
crowds  out  dollars  available  for  “lower  priority”  IT,  such  as  Manage¬ 
ment  Support. 

To  apply  the  results  of  our  study,  DoD  managers  can  disag¬ 
gregate  an  organization’s  IT  spending  allocations  into  the  four 
investment  categories  of  Innovation,  Process  Automation,  Man¬ 
agement  Support,  and  Infrastructure,  and  compare  them  with  the 
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results  in  our  study.  DoD  managers  can  also  compare  current  IT 
portfolio  investments  with  the  high-performing  agencies  or  the 
Agility-focused  agencies.  Those  who  are  in  Cost-focused  DoD 
organizations,  like  the  Defense  Finance  and  Accounting  Service, 
can  compare  their  IT  investments  with  other  Cost-focused  agencies. 

V 

O’f 


Maximizing  Federal  IT  Dollars:  A  Connection  Between  IT  Investments  and  Organizational  Performance 


April  2011 


Author  Biographies 

BG  Ennis  C.  “Jim”  Whitehead  III,  USAR 
(Ret.),  is  chief  of  External  IT  Operations 
at  the  National  Geospatial-Intelligence 
Agency  and  has  held  executive  positions 
at  major  telecommunications  corporations 
both  domestically  and  internationally.  BG 
Whitehead  received  his  BS  in  Engineering 
from  West  Point,  an  MBA  from  Harvard 
University,  a  master’s  in  Strategic  Studies 
from  the  Army  War  College,  and  a  master’s 
in  Systems  Engineering  from  The  George 
Washington  University. 

(E-mail  address:  Jim.C.Whitehead@nga.mil) 

Dr.  Shahram  Sarkani  is  a  professor  of 
Engineering  Management  and  Systems 
Engineering  at  The  George  Washington 
University.  Since  2001  he  has  served  as 
Faculty  Adviser  for  Off-Campus  Programs 
in  the  Department  of  Engineering  Manage¬ 
ment  and  Systems  Engineering.  His  current 
research  interests  include  stochastic  meth¬ 
ods  of  structural  dynamics  and  fatigue, 
fatigue  and  fracture  reliability,  structural 
safety  and  reliability,  and  smart  infrastruc¬ 
ture  systems  for  natural  hazard  mitigation. 
Dr.  Sarkani  is  a  Professional  Engineer  and 
earned  his  PhD  from  Rice  University. 

(E-mail  address:  emseocp@gwu.edu) 

Dr.  Thomas  A.  Mazzuchi  is  a  professor  of 
Engineering  Management  and  Systems 
Engineering  at  The  George  Washington 
University.  His  current  research  interests 
include  reliability  and  risk  analysis,  Bayes¬ 
ian  inference,  quality  control,  stochastic 
models  of  operations  research,  and  time 
series  analysis.  Dr.  Mazzuchi  earned  his 
PhD  from  The  George  Washington 
University. 


(E-mail  address:  emseocp@gwu.edu) 


193 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


REFERENCES 

Aral,  S.,  &  Weill,  P.  (2007).  IT  assets,  organizational  capabilities,  and  firm  performance:  How 
resource  allocations  and  organizational  differences  explain  performance  variation. 
Organization  Science,  78(5),  763-780. 

Arveson,  P.  (1999).  Translating  performance  metrics  from  the  private  to  the  public  sector 
Retrieved  from  Balanced  Scorecard  Institute  website:  http://www.balancedscorecard. 
org/BSCResources/PerformanceMeasurement/TranslatingMetrics/tabid/139/Default. 
aspx 

Brewer,  G.  A.,  &  Selden,  S.  C.  (2000).  Why  elephants  gallop:  Assessing  and  predicting 
organizational  performance  in  federal  agencies.  Journal  of  Public  Administration 
Research  and  Theory,  70(4),  685-711. 

Chief  Information  Officers  Council.  (2010,  July  1).  Statement  of  Vivek  Kundra,  Federal  Chief 
Information  Officer,  before  the  House  Committee  on  Oversight  and  Government 
Reform.  Cloud  computing:  Benefits  and  risks  of  moving  federal  IT  into  the  cloud. 
Retrieved  from  http://www.cio.gov/Documents/Vivek-Kundra-Testimony-Cloud- 
Computing_07-01-2010.pdf 

Clinger-Cohen  Act  of  1996,  40  U.S.C.  1401  et  seq.,  Pub.  L.  104-106  (1996). 

Garson,  G.  D.  (2003).  Information  technology:  Policy  and  management  issues.  Hershey,  PA: 
Idea  Group  Publishing. 

General  Accounting  Office.  (2000).  Information  technology  investment  management:  A 

framework  for  assessing  and  improving  process  maturity  (Version  1).  GAO  Report  No. 
GAO/AIMD-IO.1.23.  Retrieved  from  http://www.gao.gov/special.pubs/ail0123.pdf 

Government  Performance  and  Results  Act  of  1993,  Pub.  L.  103-62  (1993). 

Gueorguieva,  V.,  Accius,  J.,  Apaza,  C.,  Bennett,  L.,  Brownley,  C.,  Cronin,  S.,  &  Preechyanud, 

P.  (2009).  The  Program  Assessment  Rating  Tool  and  the  Government  Performance 
and  Results  Act:  Evaluating  conflicts  and  disconnections.  American  Review  of  Public 
Administration,  39(3),  225-245. 

Holmes,  D.  (2001).  eGov:  eBusiness  strategies  for  government.  Naperville,  IL:  Nicholas  Brealey. 

Kaplan,  R.  S.,  &  Norton,  D.  P.  (2005).  The  balanced  scorecard:  Measures  that  drive 
performance.  Harvard  Business  Review,  83(7/8),  172-180. 

Keyes,  J.  (2005).  Implementing  the  IT  balanced  scorecard:  Aligning  IT  with  corporate  strategy. 
Boca  Raton,  FL:  Auerbach  Publications. 

Maizlish,  B.,  &  Handler,  R.  (2005).  IT  portfolio  management  step-by-step:  Unlocking  the 
business  value  of  technology.  Hoboken,  NJ:  John  Wiley  &  Sons. 

Markowitz,  H.  (1952).  Portfolio  selection.  Journal  of  Finance,  7(1),  77-91. 

McFarlan,  F.  W.  (1981).  Portfolio  approach  to  information  systems.  Harvard  Business  Review, 
59(5),  142-150. 

Office  of  Management  and  Budget.  (2008).  Preparation,  submission,  and  execution  of  the 

budget.  OMB  Circular  No.  A-ll,  Section  53.  Retrieved  from  http://www.whitehouse.gov/ 
omb/circulars/all/current_year/s53.pdf 

Office  of  Management  and  Budget.  (2009a).  The  program  assessment  rating  tool.  Retrieved 
from  http://www.whitehouse.gov/omb/expectmore/part.html 

Office  of  Management  and  Budget.  (2009b).  ExpectMore.gov:  Frequently  asked  questions. 
Retrieved  from  http://www.whitehouse.gOv/omb/expectmore/faq.html#005 

Office  of  Management  and  Budget  (2009c).  Budget  of  the  United  States  Government. 
Retrieved  from  Government  Printing  Office  (GPO)  Access  database:  http://www. 
gpoaccess.gov/usbudget/ 

Office  of  Management  and  Budget.  (2009d).  The  FY 2008  performance  report  of 
the  federal  government.  Retrieved  from  http://www.whitehouse.gov/omb/ 
expectmore/2008  Performance,  pdf 

Office  of  Management  and  Budget.  (2010).  IT  dashboard.  Retrieved  from  http:// 
it.usaspending.gov/ 


194 


Maximizing  Federal  IT  Dollars:  A  Connection  Between  IT  Investments  and  Organizational  Performance 


April  2011 


Office  of  Personnel  Management.  (2008).  President’s  quality  award  program.  Retrieved  from 
http://www.opm.gov/pqa/past_pqa_winners/2008/index.asp 

Ostroff,  F.  (2006).  Change  management  in  government.  Harvard  Business  Review,  84(5), 
141-147. 

Peters,  G.  (1988).  Evaluating  your  computer  investment  strategy.  Journal  of  Information 
Technology,  3(3),  178-188. 

Popovich,  M.  G.,  Ed.  (1998).  Creating  high-performance  government  organizations.  San 
Francisco:  Jossey-Bass. 

Rai,  A.,  Patnayakuni,  R.,  &  Patnayakuni.  N.  (1997).  Technology  investment  and  business 
performance.  Communications  of  the  ACM,  40(7),  89-97. 

Van  Over,  D.  (2009).  Use  of  information  technology  investment  management  to  manage  state 
government  information  technology  investments.  In  A.  W.  K.  Tan  &  P.  Theodorou  (Eds), 
Strategic  Information  Technology  and  Portfolio  Management  (chap.  1).  Hershey,  PA: 
Information  Science  Reference. 

Ward,  J.  M.  (1990).  A  portfolio  approach  to  evaluating  information  systems  investments  and 
setting  priorities.  Journal  of  Information  Technology,  5(4),  222-231. 

Weill,  P.,  &  Aral,  S.  (2003).  Managing  the  IT  portfolio  (update  circa  2003).  Retrieved  from  MIT 
Sloan  Center  for  Information  Systems  Research  website:  http://cisr.mit.edu/ 

Weill,  P.,  &  Aral,  S.  (2004).  IT  savvy  pays  off.  Retrieved  from  MIT  Sloan  Center  for  Information 
Systems  Research  website:  http://cisr.mit.edu/ 

Weill,  P.,  &  Aral,  S.  (2008).  Managing  the  IT  portfolio  (update  circa  2008):  It’s  all  about  what’s 
new.  Retrieved  from  MIT  Sloan  Center  for  Information  Systems  Research  website:  http:// 
cisr.mit.edu/ 

Weill,  P.,  &  Ross,  J.  W.  (2004).  IT  governance:  How  top  performers  manage  IT  decision  rights 
for  superior  results.  Boston,  MA:  Harvard  Business  School  Press. 

Weill,  P.,  Woerner,  S.  L.,  &  Rubin,  H.  A.  (2008).  Managing  the  IT  portfolio  (update  circa  2008): 
It’s  all  about  what’s  new  (Vol.  VIII,  2B).  Retrieved  from  MIT  Sloan  Center  for  Information 
Systems  Research  website:  http://cisr.mit.edu/ 

West,  D.  M.,  &  Lu,  J.  (2009).  Comparing  technology  innovation  in  the  private  and  public 
sectors.  Retrieved  from  Brookings  Institution  website:  http://www.brookings.edu/ 
papers/2009/06_technology_west.aspx 


195 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


MOVING  TOWARD  IMPROVED 
ACQUISITION  OUTCOMES: 

THE  INTERRELATIONSHIPS 
BETWEEN  CULTURE, 
COMMITMENT  AND 
LEADERSHIP 


\  Everett  Roper 

The  impact  of  organizational  culture,  management  leadership 
style,  and  employee  commitment  on  organizational  outcomes 
has  long  been  studied,  but  no  clear  answer  exists  for  which 
concepts  most  affect  acquisition  outcomes  and  increase  orga¬ 
nizational  productivity.  A  key  contribution  of  this  study  is  the 
notion  that  they  are  interrelated  and  may  work  synergistically 
in  improving  acquisition  outcomes.  The  author  claims  that  the 
interaction  of  these  elements,  when  combined,  may  produce 
a  total  effect  that  is  greater  than  the  sum  of  their  individual 
elements.  A  conceptual  model  was  identified  and  used  as 
the  foundation  for  building  hypotheses.  Structural  Equation 
Modeling  was  used  to  analyze  the  data  gathered,  and  a  path 
diagram  was  developed  for  this  study  using  Analysis  of  Moment 
Structures  (AMOS). 


Keywords:  Organizational  Culture,  Leadership, 
Structural  Equation  Modeling,  Analysis  of  Moment 
Structures  (AMOS),  Organizational  Productivity 
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Early  studies  of  organizational  productivity  generally  begin 
with  Frederick  W.  Taylor's  (1911)  theories  of  scientific  management 
and,  more  specifically,  division  of  labor.  Taylor’s  theories  included 
the  belief  that  management’s  responsibility  was  to  plan  work,  and 
workers’  responsibility  was  to  perform  the  assigned  work  tasks. 
These  principles  were  implemented  in  many  factories  and  often 
increased  productivity;  however,  they  also  increased  the  monotony 
of  work  and  subsequently  did  little  to  improve  employee  commit¬ 
ment  or  morale.  While  Taylor  may  have  had  the  right  idea  for  the 
time,  it  can  be  argued  that  he  did  not  have  the  correct  approach 
for  today’s  environment.  The  lack  of  worker  input  and  involvement 
held  over  as  an  artifact  of  scientific  management  added  to  worker 
frustration.  Taylor’s  theory  did  not  take  into  account  external  factors 
such  as  the  leadership  style  exhibited  by  management,  relationships 
among  the  workers,  the  culture  of  the  organization,  the  motivation 
of  the  workers,  or  their  input.  Neither  did  he  consider  the  differing 
personalities  of  workers  and  managers. 

Taylor's  principles  were  developed  in  the  late  1800s,  but  are 
still  being  practiced  today.  Some  managers  are  working  to  improve 
outcomes  and  boost  productivity  without  realizing  that  they  may 
be  doing  just  the  opposite.  If,  in  an  effort  to  boost  organizational 
outcomes,  increase  revenue,  improve  customer  service,  and  drive 
increased  productivity,  they  constrain  their  employees,  do  not  seek 
their  input,  and  consequently  stifle  creative  problem  solving,  their 
methods  are  misguided.  Numerous  external  factors  are  overlooked 
by  managers  who  see  harder  or  longer  working  employees  as  the 
only  avenue  to  improvements  to  efficiency  and  productivity. 

Conceptual  Model 

The  conceptual  model  for  this  study  is  presented  in  Figure  1.  This 
model  shows  antecedents  of  factors  that  have  been  found  to  affect 
organizational  outcomes.  Arrows  are  shown  to  depict  these  rela¬ 
tionships.  Relationships  exist  between  leadership  style  and  culture, 
suggesting  that  a  manager’s  leadership  affects  an  organization’s 
culture;  between  culture  and  commitment,  suggesting  that  the  cul¬ 
ture  of  an  organization  affects  an  employee’s  level  of  commitment; 
and  between  leadership  style  and  commitment,  suggesting  that  a 
manager’s  leadership  style  directly  affects  an  employee’s  level  of 
commitment  to  the  organization— and  an  employee’s  level  of  com¬ 
mitment  affects  a  leader’s  style.  Further,  the  belief  is  that  all  these 
concepts  affect  outcomes.  As  a  result  of  this  model,  the  following 
hypotheses  were  studied. 
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FIGURE  1.  CONCEPTUAL  MODEL 
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Hypothesis  No.  1 

No  significant  relationship  exists  between  employees’  individual 
commitment  and  culture. 

Hypothesis  No.  2 

No  significant  relationship  exists  between  employees’  individual 
commitment  and  management  leadership  style. 

Hypothesis  No.  3 

No  significant  relationship  exists  between  employees’  percep¬ 
tion  of  management  leadership  style  and  culture. 

Hypothesis  No.  4 

No  significant  relationship  exists  between  employees’  perception 
of  management  leadership  style  and  organizational  commitment. 

Hypothesis  No.  5 

No  significant  relationship  exists  between  employees’  percep¬ 
tion  of  organizational  culture  and  organizational  commitment. 

Hypothesis  No.  6 

No  significant  relationship  exists  between  employees’  percep¬ 
tion  of  organizational  culture  and  management  leadership  style. 
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Definition  of  Terms 

To  situate  this  study,  an  explanation  of  the  terms  that  are  central 
to  this  study  is  needed.  The  following  definitions  were  used: 

Culture 

Wallach  (1983,  p.  29)  defined  culture  as:  the  shared  under¬ 
standing  of  an  organization’s  employees.  Wallach  labeled  three 
separate  organizational  culture  types  as  bureaucratic,  innovative, 
and  supportive.  These  types  can  be  distinguished  as  (a)  bureau¬ 
cratic-structured,  ordered,  regulated,  and  power-oriented;  (b) 
innovative— results-  and  risk-oriented;  and  (c)  supportive— collab¬ 
orative  and  relationships-oriented. 

Leadership 

For  purposes  of  this  study,  Bass’s  (1985)  definition  of  leader¬ 
ship— the  observed  effect  of  one  individual’s  ability  to  change  other 
people’s  behaviors  by  altering  their  motivations— will  be  used.  Lead¬ 
ers  are  characterized  as  one  of  three  types:  (a)  transactional— one 
who  uses  rewards  as  a  control  mechanism  to  externally  motivate; 

(b)  transformational— one  who  uses  rewards  to  increase  commit¬ 
ment  and  internally  motivate;  or  (c)  laissez-faire— one  who  offers 
no  feedback  or  support. 

Commitment 

The  definition  of  commitment  that  was  used  is:  the  strength  of 
an  individual’s  identification  with  and  involvement  in  a  particular 
organization. ..characterized  by  three  factors:  (a)  a  strong  belief  in, 
and  acceptance  of,  the  organization’s  goals  and  values;  (b)  a  readi¬ 
ness  to  exert  considerable  effort  on  behalf  of  the  organization;  and 

(c)  a  strong  desire  to  remain  a  member  of  the  organization  (Mowday, 
Porter,  &  Steers,  1982,  p.  27). 

Outcomes 

Organizational  outcomes  are  made  up  of  subsets  of  per¬ 
formance  areas,  which  are  attached  to  each  other  and  that  the 
organization  has  decided  to  maximize,  which  then  form  a  greater 
system  or  process  (Walker,  2000,  p.  1). 

Research  Methodology 


The  Organization 

The  organization  used  in  this  study  was  a  large,  high-technology 
organization  offering  services  to  the  aerospace,  energy,  and  envi¬ 
ronmental  industries.  The  total  population  in  the  business  unit  was 
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725  individuals.  The  pilot  survey  sample  consisted  of  38  individuals, 
and  the  final  survey  sample  consisted  of  164.  Employees  used  in  the 
pilot  study  did  not  participate  in  the  final  survey. 

Survey  Instruments 

Pre-established  surveys  were  used  to  obtain  measures  of  orga¬ 
nizational  culture,  management  leadership  style,  and  organizational 
commitment.  An  additional  questionnaire  was  developed  to  gather 
organizational  outcomes  from  managers  and  top  executives. 

Culture.  Wallach’s  (1983)  Organizational  Culture  Index  (OCI)  was 
used  to  assess  the  culture  of  the  organization.  The  descriptive  items 
of  the  survey  are  shown  in  Table  1.  Wallach  identified  these  culture 
indicators  as  bureaucratic,  innovative,  and  supportive.  The  OCI  was 


TABLE  1.  ORGANIZATIONAL  CULTURE  INDEX 

risk  taking  I 

results-oriented  I 

creative  I 

pressurized  I 

stimulating  I 

challenging  I 

enterprising  I 

driving  I 


collaborative  S 

trusting  S 

safe  S 

equitable  S 

personal  freedom  S 

relationships-oriented  S 

encouraging  S 

sociable  S 

structured  B 

ordered  B 

procedural  B 

hierarchical  B 

regulated  B 

established,  solid  B 

cautious  B 

power-oriented  B 
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given  to  each  employee  and  section  manager  to  rate  the  culture  in 
their  section  as  well  as  the  division  manager  or  deputy  to  rate  the 
culture  in  each  section  under  their  control. 

Commitment.  Mowday,  Porter,  and  Steers'  (1982)  Organizational 
Commitment  Questionnaire  (OCQ)  was  used  to  assess  the 
commitment  level  of  the  employee.  The  questionnaire  measured 
motivation,  intent  to  remain,  acceptance  of  goals,  and  willingness 
to  work  hard.  An  OCQ  is  classified  as  an  attitudinal  measure  of 
organizational  commitment. 

Leadership.  Bass  and  Avolio’s  Multifactor  Leadership  Questionnaire 
(MLQ)  5X  Short  Form  (1994)  was  designed  for  analyzing  leaders’ 
self-reported  leadership  styles— transformational,  transactional,  or 
laissez-faire— as  well  as  employees’  perception  of  leaders’  styles  and 
commitment  to  organizational  outcomes. 

Organizational  Outcomes.  Management  was  asked  questions 
designed  to  measure  organizational  outcomes.  Table  2  lists  the 
questions.  These  questions  were  mutually  agreed  upon  by  the 
researcher  and  the  management  of  the  organization.  Because 
of  privacy  concerns,  the  organization  was  not  in  favor  of  a  more 
extensive  list  of  questions  recommended  by  the  researcher. 


TABLE  2.  ORGANIZATIONAL  OUTCOMES 


1. 

Percentage  of  tasks  completed  on  schedule 

2. 

Average  number  of  critical  events  found  during  design  review 

2a.  Additional  time  added  to  schedule  as  a  result? 

2b.  Significant  budget  increase  as  a  result? 

2c.  Personnel  additions  as  a  result? 

3. 

Number  of  proposal  iterations 

4. 

In  the  past  year,  how  many  employees  have  left  the  Section? 

5. 

In  the  past  year,  how  many  employees  were  replaced  in  the 

Section? 

6. 

In  the  past  year,  how  many  employees  were  added  to  the  Section? 

Reliability  and  Unidimensionality.  A  factor  analysis  was  performed 
on  each  questionnaire  to  analyze  interrelationships  among  the 
questions  as  well  as  to  explain  the  variables  in  terms  of  their 
underlying  common  factors.  As  a  result,  a  pilot  study  and  subsequent 
factor  analysis  for  each  questionnaire  resulted  in  the  elimination  of 
variables  that  either  seemed  inconsistent  with  related  responses  or 
appeared  to  be  ambiguous  to  those  taking  the  survey.  Throughout 
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this  analysis,  validity  values  less  than  0.6  were  eliminated.  During  the 
analysis,  initial  results  did  not  provide  a  clear,  simple  interpretation 
of  the  data.  As  a  result  of  these  issues,  a  Varimax  rotation1  was 
applied  and  several  iterations  were  run  after  eliminating  variables 
that  did  not  meet  the  loading  threshold.  All  of  the  following  results 
show  final  computational  data  after  rotations  are  applied.  For  each 
analysis,  the  V’s  in  the  far  right  column  denote  variables  that  should 
not  be  discarded  since  they  load  on  only  one  factor  and  are  above 
the  0.6  validity  threshold.  To  support  the  number  of  factors  for  each 
analysis,  the  Cattell  (1966)  Scree  Test  was  also  used. 

Commitment  Factor  Analysis.  Table  3  shows  the  commitment  factor 
analysis  output.  All  of  the  variables  loaded  on  separate  factors  with 

TABLE  3.  COMMITMENT  FACTOR  ANALYSIS-FINAL 


Component 


1 

2 

3 

comml 

.957 

.238 

-.091 

comm2 

-.319 

.909 

-.182 

comm5 

.049 

.620 

.679 

comm6 

-.319 

.909 

-.182 

comm8 

.957 

.238 

-.091 

comml2 

.454 

-.183 

.592 

comml3 

-.312 

.164 

.602 

comm14 

.957 

.238 

-.091 
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a  clear  division  among  them.  With  the  exception  of  comm12,  all  of 
the  variables  exceeded  the  threshold  value.  However,  comm12  was 
not  eliminated  since  it  was  within  a  few  thousandths  of  the  0.6 
threshold  value.  As  a  result,  these  variables  were  used  in  the  final 
statistical  analysis  because  of  their  high  loading  and  significance  to 
the  corresponding  factor  as  highlighted  in  the  table. 

From  the  commitment  scree  plot  in  Figure  2,  it  can  be  seen 
that  the  first  three  factors  are  worth  retaining  in  the  analysis.  This 
is  consistent  with  the  number  of  factors  in  the  final  analysis  results 
in  Table  3. 

Culture  Factor  Analysis.  Table  4  shows  the  culture  factor  analysis 
output.  From  the  culture  scree  plot  in  Figure  3,  it  can  be  seen  that 
the  first  four  factors  are  worth  retaining  in  the  analysis.  This  was 
one  factor  more  than  the  number  of  factors  determined  from  the 
final  analysis  results  in  Table  4,  which  showed  only  three  factors 
being  retained. 

TABLE  4.  CULTURE  FACTOR  ANALYSIS-FINAL 


Component 


1 

2 

3 

cult2 

.678 

-.200 

-.150 

cult7 

.700 

-.071 

-.458 

cult8 

.446 

-.071 

.706 

cultIO 

.431 

.621 

.045 

cult13 

-.483 

-.277 

.596 

cult14 

-.415 

.664 

.439 

cult16 

.625 

-.542 

.046 

cult22 

.509 

.649 

-.219 

Leadership  Factor  Analysis.  Table  5  shows  the  initial  leadership 
factor  analysis  output.  These  four  variables  were  used  in  the  final 
statistical  analysis.  From  the  leadership  scree  plot  in  Figure  4,  it 
can  be  seen  that  the  first  four  factors  were  worth  retaining  in  the 
analysis.  This  was  inconsistent  with  the  number  of  factors  in  the 
final  analysis  results  in  Table  5,  which  showed  only  one  factor  being 
retained.  This  caused  some  concern,  but  the  scree  test  could  be  in 
error  since  it  suffers  from  subjectivity  and  ambiguity  (Hayton,  Allen, 
&  Scarpello,  2004). 

Of  the  many  coefficients,  Cronbach’s  alpha  (1951)  is  probably 
the  best  known.  Cronbach’s  alpha  is  regarded  as  the  lower  bound 
on  reliability  for  a  set  of  congeneric  measures.  It  assumes  each  of 
the  items  within  the  scale  contributes  equally  to  the  underlying  trait. 
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TABLE  5.  LEADERSHIP  FACTOR  ANALYSIS-FINAL 


Iead5 

.873 

leadIO 

.836 

Iead15 

.876 

Iead22 

.923 

FIGURE  4.  LEADERSHIP  SCREE  PLOT 
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The  final  Cronbach  alpha  values  for  each  construct  in  this  study 
are  reported  in  Table  6.  The  measures  were  relatively  homogeneous 
for  the  construct  they  purport  to  measure.  Typically,  reliabilities 
greater  than  0.7  are  considered  adequate  for  measurement  analysis 
(Nunnally,  1978).  However,  while  the  bureaucratic  culture  alpha  value 
was  reasonably  low,  it  was  accepted  since  attempts  to  increase  its 
value  reduced  the  alpha  values  of  the  other  factors.  In  the  end,  the 
values  shown  reflect  the  most  stable  values. 


TABLE  6.  SURVEY  INSTRUMENT  ALPHA  VALUES 


Factor 

Number  of 
Items 

Alpha 

Culture  (Bureaucratic) 

8 

0.513 

Culture  (Innovative) 

8 

0.695 

Culture  (Supportive) 

8 

0.811 

Commitment 

8 

0.64 

Leadership 

4 

0.90 

Structural  Equation  Modeling 

This  study  used  structural  equation  modeling  for  statistical 
analysis  of  data.  Structural  equation  modeling  (SEM)  is  a  statistical 
technique  that  integrates  path  and  factor  analysis.  Path  analysis, 
a  subset  of  SEM,  deals  only  with  measured  variables,  and  is  the 
statistical  technique  used  to  examine  causal  relationships  between 
two  or  more  variables. 

After  a  thorough  review  of  the  literature,  it  was  determined  that 
leadership,  culture,  and  individual  commitment  all  have  a  direct 
effect  on  organizational  outcomes.  A  more  extensive  literature 
review  also  revealed  that  leadership  may  affect  culture  and  com¬ 
mitment  while  culture  may  only  affect  commitment.  Figure  5  shows 
the  subsequent  path  diagram  for  these  relationships  that  were  cre¬ 
ated  using  Analysis  of  Moment  Structures,  or  AMOS.2  As  required 
by  SEM,  the  measured  variables  are  indicated  by  rectangles,  latent 
variables  by  ellipses,  and  error  terms  by  circles.  The  error  terms 
represent  residual  variances  within  variables  not  accounted  for  by 
pathways  hypothesized  in  the  model. 

With  six  connections  and  using  a  ratio  of  20:1  as  a  guide,  at 
least  120  samples  were  needed  to  adequately  support  the  statisti¬ 
cal  precision  of  the  results.  The  final  results  for  this  study  contained 
164  samples. 
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Results 


Analysis 

Numerous  tests  exist  for  assessing  how  well  a  model  matches 
the  observed  data.  Chi-square  is  the  most  common  goodness-of-fit 
measure.  In  a  full  model,  there  is  a  direct  path  from  each  variable  to 
each  other  variable.  When  one  or  more  paths  are  missing,  a  reduced 
model  is  obtained.  In  this  study,  an  analysis  was  performed  to  see 
which  model  is  better. 

The  Chi-square  value  is  72.2  with  44  degrees  of  freedom  and 
a  p-value  equal  to  0.005.  Since  this  p-value  does  not  exceed  the 
alpha  value  of  0.05,  the  null  hypothesis  is  rejected,  indicating  that 
the  model  does  not  fit  the  data  adequately. 

It  should  be  noted,  however,  that  problems  with  Chi-square  are 
known  to  exist.  The  main  drawback  with  the  Chi-square  test  is  that 
it  is  sensitive  to  sample  size,  becoming  more  and  more  likely  to 
reject  the  null  hypothesis  as  the  sample  size  increases.  This  is 
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because  as  the  sample  size  increases,  the  Chi-square  statistic  has  a 
tendency  to  indicate  a  significant  probability  level  whereas  a 
decrease  in  the  sample  size  results  in  a  commensurate  decrease  in 
the  statistic  to  nonsignificant  levels  (Schumacker  &  Lomax,  2004). 
Therefore,  additional  testing  was  done  before  drawing  conclusions 
on  model  fit.  One  corroborating  test  of  model  fit  is  provided  by  the 
Root  Mean  Square  Error  of  Approximation  (RMSEA)  statistic— a 
measure  of  fit  introduced  by  Steiger  and  Lind  (1980).  The  RMSEA 
estimates  lack  of  fit  compared  to  the  full  model.  RMSEA  differs  from 
the  Chi-square  test  in  that  it  is  sensitive  to  the  number  of  parameters 
estimated  and  relatively  insensitive  to  sample  size.  The  AMOS  4.0 
User’s  Guide  (Arbuckle,  1999)  suggests  that  an  RMSEA  value  of  0.05 
or  less  indicates  a  close  fit  of  the  model,  and  an  adequate  fit  exists 
if  RMSEA  is  less  than  or  equal  to  0.08.  MacCallum,  Browne,  and 
Sugawara  (1996)  added  that  RMSEA  values  ranging  from  0.08  to 
0.10  indicate  mediocre  fit,  and  those  greater  than  0.10  indicate  poor 
fit.  LO  90  and  HI  90  values,  as  shown  in  Table  7,  indicate  90  percent 
confidence  limits  on  the  coefficient  while  PCLOSE3  tests  the  null 
hypothesis  that  RMSEA  is  no  greater  than  0.05.  MacCallum  et  al. 
(1996)  contend  that  when  a  small  RMSEA  exists,  with  a  wide  con¬ 
fidence  interval,  it  can  be  concluded  that  the  estimated  discrepancy 
value  is  quite  imprecise.  This  then  negates  any  possibility  of  deter¬ 
mining  an  accurate  degree  of  fit  in  the  population.  In  contrast,  a 
very  narrow  confidence  interval  would  argue  for  good  precision  of 
the  RMSEA  value  in  reflecting  model  fit  in  the  population  (MacCal¬ 
lum  et  al.,  1996). 


TABLE  7.  RMSEA  (INITIAL) 


Model 

RMSEA 

LO  90 

HI  90 

PCLOSE 

Default  model 

0.063 

0.035 

0.088 

0.204 

Independence  model 

0.322 

0.305 

0.34 

0 

As  noted  earlier,  PCLOSE  is  a  statistical  significance  test  of  the 
RMSEA  and  measures  the  p  value  by  testing  the  null  that  RMSEA  is 
no  greater  than  0.05.  As  shown  in  Table  7,  the  RMSEA  value  of  this 
model  is  0.063,  indicating  that  the  estimate  is  adequate.  Evidence 
affirms  that  the  estimate  is  correct  since  PCLOSE  is  0.204,  which  is 
greater  than  the  0.05  alpha  value.  Some  experts  suggest  that  the 
PCLOSE  value  should  be  greater  than  0.5;  however,  the  0.209  value 
was  accepted  since  it  is  greater  than  the  alpha  value.  As  a  result, 
the  null  hypothesis  was  not  rejected,  RMSEA  is  greater  than  0.05, 
and  thus  it  was  concluded  that  the  model  fits  the  data  adequately. 
Additionally,  the  90  percent  confidence  of  the  RMSEA  is  within  the 
bounds  of  0.035  and  0.088.  The  upper  bound  of  the  confidence 
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interval  was  fractionally  higher  than  the  suggested  cutoff  of  0.08, 
but  this  was  considered  mediocre  according  to  MacCallum  et  al. 
(1996).  Overall,  given  that  (a)  the  RMSEA  point  estimate  is  adequate, 
i.e.,  0.063  <  0.08;  (b)  the  RMSEA  point  estimate  is  within  the  90 
percent  confidence  interval;  and  (c)  the  probability  value  associated 
with  this  test  of  close  fit  is  PCLOSE  =  0.204,  it  was  concluded  that 
the  model  provides  an  adequate  fit  of  the  data. 

To  establish  greater  confidence,  an  additional  goodness-of-fit 
test  was  conducted.  The  Expected  Cross-Validation  Index  (ECVI)  is 
an  approximation  of  the  goodness-of-fit  that  the  estimated  model 
would  achieve  in  another  sample  of  the  same  size.  It  takes  into 
account  the  actual  sample  size  and  the  difference  that  could  be 
expected  in  another  sample.  The  ECVI  also  takes  into  account  the 
number  of  estimated  parameters  for  both  the  structural  and  mea¬ 
surement  models.  Application  of  the  ECVI  assumes  a  comparison 
of  models  whereby  an  ECVI  index  is  computed  for  each  model,  and 
then  all  ECVI  values  are  placed  in  rank  order.  The  model  having  the 
smallest  ECVI  value  exhibits  the  greatest  potential  for  replication 
(Byrne,  2001). 

In  assessing  the  ECVI  results  for  the  model  presented  here  (the 
default  model),  as  shown  in  Table  8  the  ECVI  value  of  0.843  for  the 
initial  model  is  compared  with  the  saturated  model  (ECVI  =  0.939) 
and  the  independence  model  (ECVI  =  6.313).  A  saturated  model 
perfectly  fits  the  data  because  it  has  as  many  parameters  as  there 
are  values  to  be  fit.  An  independence  model  on  the  other  hand  is 
one  in  which  two  (or  more)  random  variables  are  independent  of 
one  another.  Given  the  lower  ECVI  value  for  the  default  model, 
compared  with  both  the  independence  and  saturated  models,  the 
conclusion  is  that  it  represents  the  best  fit  to  the  data.  The  precision 
of  this  estimated  ECVI  value  can  also  be  taken  into  account  by 
examining  the  confidence  intervals,  which  range  from  0.724  to 
1.009.  Taken  together,  these  results  suggest  that  the  model  provides 
a  good  fit  and  represents  a  reasonable  approximation  to  the  popu¬ 
lation.  This  conclusion  supports  the  findings  of  the  RMSEA. 

TABLE  8.  EXPECTED  CROSS-VALIDATION  INDEX  (INITIAL) 


Model 

ECVI 

LO  90 

HI  90 

MECVI 

Default  model 

0.843 

0.724 

1.009 

0.874 

Saturated  model 

0.939 

0.939 

0.939 

1.013 

Independence  model 

6.313 

5.713 

6.959 

6.335 

Next,  we  examined  the  reliability  of  the  measures.  Bollen  (1989) 
suggests  that  the  Squared  Multiple  Correlation  is  an  adequate 
measure  for  doing  so.  Fornell  and  Larcker  (1981)  suggest  that  the 
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TABLE  9.  SQUARED  MULTIPLE  CORRELATIONS  (INITIAL) 

_ Estimate 

leadership  0 

culture  0 

commitment  0.089 

comm6  0.928 

cult9  -0.345 

comml2  0.165 

Iead22  0.996 

Iead5  0.772 

org  outcomes  0.768 
comm7  0.351 

comm4  0.274 

cultll  0.352 

cultl  0.717 

Ieadl5  0.902 

leadlO  0.785 


magnitude  of  this  coefficient  should  be  greater  than  0.5,  which 
implies  that  more  than  50  percent  of  the  variance  of  the  item  is 
related  to  what  is  being  measured.  The  squared  multiple  correla¬ 
tion  coefficients  are  shown  in  Table  9.  The  coefficients  that  meet 
the  0.5  threshold  recommended  by  Fornell  and  Larcker  (1981)  are 
shown  as  well  as  the  coefficients  that  are  below  the  threshold.  The 
coefficients  below  the  threshold  were  left  in  the  model  because 
removing  them  caused  model  instability  in  other  significance  tests. 
Their  inclusion  in  the  model  allowed  for  the  best  fit. 

Next,  we  examined  the  regression  weights  to  determine  if  the 
coefficients  are  significant.  The  regression  coefficients  represent 
the  amount  of  change  in  the  dependent  variable  for  each  one  unit 
of  change  in  the  variable  predicting  it.  In  Table  10,  culture  increases 
0.013  units  for  each  1.0  unit  increase  in  leadership.  The  table  displays 
the  estimate,  its  standard  error  (S.E.),  and  the  estimate  divided  by 
the  standard  error  (C.R.).  The  p-value  tests  the  null  hypothesis  that 
the  covariance  between  two  variables  is  zero  in  the  population  from 
which  this  sample  was  drawn. 

The  regression  coefficients  in  this  model  were  found  to  be 
significant  with  the  exception  of  the  culture-leadership  and 
commitment-culture  relationships.  Of  greatest  concern  was  the 
culture-leadership  relationship  p-value  of  0.868,  which  is  far  beyond 
the  0.05  alpha  level.  Numerous  model  revisions  were  performed  in 
an  effort  to  reduce  this  value  with  no  success.  However,  after  further 
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TABLE  10.  REGRESSION  WEIGHTS  (INITIAL) 


Estimate 

S.E. 

C.R. 

P 

Label 

culture 

<- 

Leadership 

0.013 

0.076 

0.167 

0.868 

par 10 

commitment 

<r 

Culture 

-0.172 

0.1 

-1.724 

0.085 

par 9 

commitment 

<r 

Leadership 

0.264 

0.099 

2.66 

0.008 

par 11 

leadlO 

<- 

Leadership 

0.889 

0.045 

19.785 

*** 

par l 

Ieadl5 

<r 

Leadership 

1.073 

0.035 

30.283 

*** 

par 2 

cultl 

Culture 

0.598 

0.078 

7.638 

*** 

par 3 

cultll 

<r 

Culture 

0.374 

0.057 

6.579 

*** 

par 4 

comm4 

<r 

Commitment 

0.581 

0.082 

7.101 

*** 

par 5 

comm7 

<r 

Commitment 

0.557 

0.072 

7.733 

*** 

par 6 

org  outcomes 

<r 

Leadership 

1 

org  outcomes 

<r 

Commitment 

1 

org  outcomes 

<- 

Culture 

1 

Iead5 

<r 

Leadership 

0.894 

0.047 

19.067 

*** 

par 7 

Iead22 

<r 

Leadership 

1 

comml2 

<r 

Commitment 

0.28 

0.053 

5.304 

*** 

par 8 

cult9 

<r 

Culture 

1 

comm6 

<r 

Errcomm6 

0.291 

0.148 

1.969 

0.049 

par 13 

comm6 

<- 

Commitment 

1 

research,  a  change  to  the  model  was  considered.  It  was  determined 
that  the  link  between  culture  and  leadership  be  reversed,  implying 
that  the  commitment  level  of  employees  affects  the  leadership  style 
that  the  manager  portrays  rather  than  management  leadership 
style  affecting  the  commitment  of  the  employees  as  the  literature 
suggests.  In  an  attempt  to  stabilize  the  model,  this  change  was 
reluctantly  made.  The  path  model  was  changed  to  reflect  this  direc¬ 
tional  change  in  the  commitment-leadership  relationship,  as  shown 
in  Figure  6.  The  revised  model  was  re-run,  and  the  new  regression 
weights  were  analyzed. 

As  shown  in  Table  11,  the  regression  results  in  the  revised  model 
show  a  drastically  reduced  p-value  of  the  commitment-leadership 
relationship  to  far  below  the  alpha  level.  However,  the  culture-lead¬ 
ership  relationship  increased  to  0.461,  which  was  much  higher  than 
the  previous  value  of  0.085.  As  a  result,  the  relationship  became 
nonsignificant.  After  several  modifications  and  re-analyses,  it  was 
finally  concluded  that  this  refined  model  would  provide  the  most 
stable  results.  The  significant  paths  are  shown  in  Figure  7.  To  verify 
this  revised  model,  all  of  the  previous  analyses  were  redone  using 
this  revised  model. 
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FIGURE  6.  AMOS  PATH  DIAGRAM  (REVISED) 


The  Chi-square  value  was  reduced  from  72.189  in  the  initial 
model  to  68.529  and  the  probability  level  from  0.005  to  0.01. 
Since  this  p-value  does  not  exceed  the  alpha  value  of  0.05,  the  null 
hypothesis  is  rejected.  The  conclusion  is  the  revised  model  does 
not  provide  an  adequate  fit  of  the  data.  This  same  conclusion  was 
reached  in  the  initial  model  and,  once  again,  additional  tests  were 
conducted  to  verify  model  fit  as  was  done  for  the  initial  model. 

As  shown  in  Table  12,  the  RMSEA  value  in  the  revised  model 
was  reduced  from  0.063  to  0.058,  which  is  slightly  above  the  0.05 
criterion  for  a  close  fit.  Thus,  the  value  was  considered  to  be  ade¬ 
quate  and  provided  greater  confidence  that  this  estimate  is  correct 
since  PCLOSE  is  0.288.  The  hypothesis  was  not  rejected  since  the 
p-value  is  greater  than  the  0.05  level  of  confidence.  It  is  concluded 
that  RMSEA  is  greater  than  0.05.  It  is  concluded  that  the  model 
fits  the  data  adequately.  Additionally,  the  90  percent  confidence 
of  the  RMSEA  is  within  the  upper  bounds  of  0.029  and  0.084.  The 
upper  bound  of  the  confidence  interval  is  fractionally  higher  than 
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TABLE  11.  REGRESSION  WEIGHTS  (REVISED  MODEL) 


Estimate 

S.E. 

C.R. 

P 

Label 

commitment 

<r 

leadership 

0.269 

0.098 

2.73 

0.006 

par 10 

culture 

<r 

leadership 

0.056 

0.076 

0.737 

0.461 

par 9 

culture 

<r 

commitment 

-0.159 

0.06 

-2.637 

0.008 

par 13 

leadIO 

<r 

leadership 

0.889 

0.045 

19.778 

*** 

par 1 

Iead15 

<- 

leadership 

1.072 

0.035 

30.273 

*  *  * 

par 2 

cultl 

<r 

culture 

0.599 

0.079 

7.567 

*  *  * 

par 3 

cultll 

<r 

culture 

0.369 

0.057 

6.502 

*  *  * 

par 4 

comm4 

<r 

commitment 

0.578 

0.081 

7.109 

*  *  * 

par 5 

comm7 

<r 

commitment 

0.553 

0.071 

7.752 

*  *  * 

par 6 

org  outcomes 

<r 

leadership 

1 

org  outcomes 

<r 

commitment 

1 

org  outcomes 

<r 

culture 

1 

Iead5 

<r 

leadership 

0.894 

0.047 

19.06 

*  *  * 

par 7 

comm12 

<r 

commitment 

0.278 

0.052 

5.309 

*  *  * 

par 8 

cult9 

<r 

culture 

1 

comm6 

<- 

errcomm6 

0.275 

0.15 

1.83 

0.067 

par 12 

comm6 

<r 

commitment 

1 

Iead22 

<r 

leadership 

1 

FIGURE  7.  SIGNIFICANT  PATHS 


Leadership 

c 

Individual 

lommitmen 

t 

213 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


214 


TABLE  12.  RMSEA  (REVISED  MODEL) 


Model 

RMSEA 

LO  90 

HI  90 

PCLOSE 

Default  model 

0.058 

0.029 

0.084 

0.288 

Independence  model 

0.322 

0.305 

0.34 

0 

TABLE  13.  SQUARED  MULTIPLE  CORRELATIONS  (REVISED 
MODEL) 


Estimate 

leadership 

0 

commitment 

0.067 

culture 

0.025 

Iead22 

0.996 

comm6 

0.934 

cult9 

-0.362 

comm12 

0.16 

Iead5 

0.772 

org  outcomes 

0.77 

comm7 

0.341 

comm4 

0.267 

cultll 

0.351 

cultl 

0.732 

Ieadl5 

0.902 

leadIO 

0.785 

the  suggested  cutoff  of  0.08— considered  mediocre  according  to 
MacCallum  et  al.  (1996).  Given  that  (a)  the  RMSEA  point  estimate  is 
0.058  <  0.08,  (b)  the  RMSEA  point  estimate  is  within  the  90  percent 
confidence  interval,  and  (c)  the  probability  value  associated  with 
this  test  of  close  fit  is  PCLOSE  =  0.288,  the  conclusion  is  that  the 
initially  hypothesized  model  provides  an  adequate  fit  of  the  data. 

The  Squared  Multiple  Correlation  values  in  Table  13  of  the  revised 
model  noted  minor  changes.  Some  coefficients  meet  the  standard 
set  by  Fornell  and  Larcker  (1981)  while  others  were  below  the  0.5 
threshold,  but  were  left  in  the  model  because  removing  them 
caused  model  instability  in  other  significant  tests.  Their  inclusion  in 
the  model  allowed  for  the  best  fit. 

Discussion  and  Conclusions 

The  first  significant  relationship  among  the  data  collected  for 
this  study  demonstrated  that  leadership  does  in  fact  affect  com- 
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mitment.  It  was  also  determined  that  commitment  affects  culture. 
Both  of  these  conclusions  are  not  surprising  since  they  support  the 
findings  in  the  research  literature,  which  demonstrated  that  there  is 
a  link  between  these  concepts.  The  effects  were  both  positive  and 
significant  implying  that  the  leader  has  the  ability  to  influence  the 
commitment  of  the  employees  and  that  the  commitment  level  of 
the  employees  affects  the  culture  of  the  organization.  The  use  of 
regression  analysis  supported  these  findings. 

Leadership,  however,  was  found  to  have  no  significant  impact 
on  culture.  This  was  unexpected.  Based  on  the  literature,  it  was 
believed  that  leadership  styles  could  have  a  significant  effect  on 
establishing  the  culture  of  an  organization  because  of  their  per¬ 
ceived  interconnection.  As  a  result,  further  study  is  needed  to 
determine  the  accuracy  of  this  conclusion,  necessitating  the  need 
to  check  to  see  if  these  results  are  consistent  for  broader  ranges 
of  conditions. 


Managerial  Implications 

The  implications  for  managers  are  many.  A  common  assump¬ 
tion  is  that  the  culture  within  the  organization  is  directly  linked 
to  the  outcomes  of  an  organization,  and  that  changes  to  culture 
traits  will  impact  effectiveness.  However,  this  study  provided  some 
preliminary  evidence  that  this  presumption  alone  may  not  be  true. 
Evidence  suggests  that  different  cultures  that  are  sensitive  to  either 
external  conditions  or  internal  conditions  may  have  a  different 
impact  on  organizational  outcomes.  An  externally  focused  culture 
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type  impacts  revenue,  sales  growth,  and  market  share.  It  is  a  cul¬ 
ture  that  brings  together  the  elements  of  mission  and  adaptability. 
It  is  goal  sensitive,  but  it  is  also  ready  to  quickly  react  to  market 
or  consumer  fluctuations  (Hastings  &  Potter,  2004).  In  an  inter¬ 
nally  focused  culture,  outcomes  are  significantly  influenced  by  the 
extent  to  which  a  leader  is  supportive  of  followers  and  includes 
followers  in  decision-making  processes.  Managers  must  be  aware 
of  this  and  manage  both  environments  effectively  to  see  outcome 
improvements. 

Although  the  results  of  this  study  do  not  confirm  an  associa¬ 
tion  between  leadership  and  culture,  the  literature  shows  that  such 
a  relationship  does  in  fact  exist  and  that  associations  between 
leadership  styles  and  organizational  outcomes  are  mediated  by 
some  form  of  organizational  culture.  In  agreement  with  the  litera¬ 
ture  findings,  the  author  contends  that  potential  solutions  to  the 
difficulties  associated  with  changing  organizational  culture  may 
involve  focusing  on  leadership  style.  While  managing  culture  is  at 
best  difficult,  changes  to  leadership  styles  may  allow  changes  to 
culture  to  be  more  easily  achieved.  Thus,  an  effective  manager  can 
influence  and  manage  culture  since  the  impact  of  poor  leadership 
skills  demonstrated  by  leaders  will  have  an  effect  on  organizational 
outcomes  and  subsequently  culture.  Thus,  training  in  organizational 
leadership  is  needed.  Leadership  training  results  in  many  benefits 
for  both  managers  and  employees. 

Additionally,  organizations  have  to  find  new  ways  to  create  a 
committed  workforce.  Managers  need  to  understand  the  concept 
of  commitment  and  which  behaviors  are  displayed  by  employees 
committed  to  the  organization  (Coetzee,  Martins,  Basson,  &  Muller, 
2006).  Successful  organizations  today  must  have  managers  who 
motivate  and  inspire  their  employees.  Successful  managers  must 
see  themselves  not  just  as  bosses,  but  as  performance  coaches.  A 
manager  must  be  able  to  provide  employee  training,  help  employ¬ 
ees  enhance  their  careers,  and  mentor  them  to  become  the  best 
they  can  be. 

Theoretical  Implications 

Overall  the  results  supported  most  of  the  literature  findings  of 
the  interrelationships  between  the  engineering  concepts;  however, 
conclusive  evidence  could  not  be  obtained  on  the  effect  that  these 
relationships  had  on  organizational  outcomes  due  to  the  lack  of 
organizational  outcome  data.  Only  10  percent  of  the  responses 
were  returned.  This  data  proved  to  be  inadequate  in  drawing  con¬ 
clusions  on  organizational  outcomes.  This  low  response  was  mainly 
attributed  to  management’s  concern  for  privacy.  While  the  study 
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did  receive  support  from  top  management,  it  appears  that  mid-level 
managers  were  not  comfortable  providing  such  confidential  data. 

Conclusions 

The  results  of  this  study  provided  initial  support  for  the  pre¬ 
sumed  relationships  among  the  conceptual  model  presented  in  the 
study  and  therefore  do  validate  a  number  of  ideas  for  organizations 
interested  in  knowing  how  to  improve  organizational  outcomes; 
however,  these  findings  need  to  be  further  validated  with  addi¬ 
tional  studies  on  a  more  diverse  population.  Organizations  and 
managers  may  infer  from  this  study  that  a  linkage  exists  among 
several  of  the  engineering  management  concepts  presented.  The 
results  suggest  that  a  relationship  exists  between  leadership,  posi¬ 
tive  individual  commitment,  and  the  right  culture.  What  remains  to 
be  proven  is  if  this  may  indeed  lead  to  heightened  organizational 
outcomes.  Thus,  while  these  concepts  have  long  been  studied  and 
supported  in  popular  management  literature,  a  key  contribution  of 
this  study  is  the  notion  that  they  are  interrelated  and  that  they  may 
work  synergistically  in  their  effect  upon  organizational  outcomes 
in  high-technology  organizations.  After  careful  consideration,  this 
researcher  hopes  that  this  study  may  shed  light  on  some  new 
linkage  between  leadership,  commitment,  and  culture;  and  recom¬ 
mends  that  future  studies  increase  the  number  of  subjects  from 
various  types  of  organizations. 
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ENDNOTES 

1.  Varimax  rotation  is  often  used  in  surveys  to  see  how  groupings  of  questions  (items) 
measure  the  same  concept.  In  statistics,  a  Varimax  rotation  is  a  change  of  coordinates 
used  in  principal  component  analysis  and  factor  analysis  that  maximizes  the  sum  of  the 
variances  of  the  squared  loadings. 

2.  AMOS  (Analysis  of  Moment  Structures)  is  an  add-on  module  for  a  computer  program 
called  SPSS  (originally,  Statistical  Package  for  the  Social  Sciences).  AMOS  is  designed 
primarily  for  structural  equation  modeling,  path  analysis,  and  covariance  structure 
modeling.  It  features  an  intuitive  graphical  interface  that  allows  the  analyst  to  specify 
models  by  drawing  them.  It  also  has  a  built-in  bootstrapping  routine  and  superior 
handling  of  missing  data.  It  reads  data  from  a  number  of  sources,  including  MS  Excel 
spreadsheets  and  SPSS  databases. 

3.  p  of  Close  Fit  (PCLOSE)— The  null  hypothesis  is  that  the  RMSEA  is  .05,  a  close-fitting 
model.  The  p  value  examines  the  alternative  hypothesis  that  the  RMSEA  is  greater  than 
.05.  So  if  the  p  is  greater  than  .05,  then  it  is  concluded  that  the  fit  of  the  model  is  “close.” 
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Review: 

With  painstaking  detail  and  an  eye  for  the  “road  not  taken,”  Maggie  Mort 
explores  the  history  of  Britain’s  Trident  submarine  program,  one  of  the  more 
complex  defense  acquisition  efforts  undertaken  by  the  UK.  Initiated  in  the 
waning  days  of  the  Cold  War,  the  Trident  program  breathed  new  life  into  the 
shipbuilding  community  of  Barrow,  but  short-  and  long-term  decisions  made 
by  the  shipyard’s  parent  company,  combined  with  political  jockeying  by  union 
leadership  and  the  Barrow  Alternative  Employment  Committee  (BAEC),  resulted 
in  the  slow  decline  of  both  the  shipyard  and  the  surrounding  community. 

At  the  heart  of  Mort’s  exploration  of  the  Trident  is  the  concept  of  “enroll¬ 
ment”:  the  bureaucratic  and  ideological  work  of  pulling  together  labor, 
management,  and  machines  to  create  a  complex  network  to  build  submarines. 
She  examines  the  decisions  of  Vickers  Shipbuilding  and  Engineering  Ltd  (VSEL) 
between  the  late  1980s  and  1998,  which  included  a  transition  from  diverse 
manufacturing  activities  to  a  focus  on  the  defense  sector  as  a  “core  business.” 
As  VSEL  was  taking  these  steps,  the  BAEC— a  collective  of  anti-nuclear  and  com¬ 
munity  activists  who  joined  forces  with  union  employees  of  VSEL— was  pushing 
to  broaden  the  work  of  the  shipyard  and  refocus  workers’  energies  away  from 
the  defense  business.  Mort  adopts  the  employees’  point  of  view  in  exploring 
why  BAEC  ultimately  lost  out  in  its  struggles,  despite  compelling  economic 
arguments  in  favor  of  diversification  that  were  supported  by  key  actors  in  the 
regional  government. 

What  the  employees  feared,  and  what  VSEL  had  to  manage  as  the  Trident 
program  proceeded,  was  “disenrollment”— the  flip  side  of  the  enrollment  coin. 
Disenrollment  takes  the  form  of  labor  retrenchment,  broken  ties  with  supply 
chain  vendors,  and  management  decisions  that  have  the  effect  of  shaping  future 
business  opportunities— actions  that  appeared,  from  the  standpoint  of  VSEL  and 
elements  of  the  shipyard  tradesmen,  to  make  perfect  sense.  The  power  struggles 
that  took  place  in  and  around  Barrow  were  not  simply  “public  versus  private”  or 
“establishment  versus  activist”  confrontations;  Mort’s  nuanced  analysis  of  the 
Trident  program  shows  how  allegiances  can  change;  decisions  about  technol¬ 
ogy  can  shape  a  workforce;  and  local  politics— and  the  insecurities  expressed 
by  a  working  community— can  derail  otherwise  rational  economic  development 
initiatives. 

The  acquisition  professional  will  find  this  case  study  interesting  for  a  number 
of  reasons.  The  overarching  theme  of  the  book— uncertainty— deserves  con¬ 
sideration.  The  Trident  program  was  marked  by  uncertainty,  and  that  sense  of 
uncertainty  colored  the  interactions  of  the  BAEC,  VSEL,  and  various  union  and 
community  leaders.  As  the  Department  of  Defense  faces  a  wave  of  reductions 
in  the  early  2010s,  DoD  acquisition  professionals  can  take  this  book  as  evidence 
that  acquisition  decisions  are  not  made  in  a  vacuum;  ultimately,  just  like  politics, 
all  programs  are  local. 
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